1 of 24

Provisioning

This page describes how to setup connections to Kafka and other services and have changes applied automatically for the Lenses Agent.

Overview

Learn about provisioning.

Learn how to connect the Agent to HQ.

Kafka

Learn how to connect the Agent to Kafka.

Schema Registries

Learn how to connect the Agent to Schema Registries.

Kafka Connect

Learn how to connect the Agent to Kafka Connect Clusters.

Zookeeper

Learn how to connect the Agent to Zookeeper.

AWS

Learn how to connect the Agent to AWS.

Alert & Auditing Integrations

Learn how to connect the Agent to Alert & Audit Integrations.

JMX

Learn how to connect the Agent to JMX for Kafka, Schema Registries, Kafka Connect and others.

Overview

This page describes an overview of Lenses Agent Provisioning.

As of version 6.0 the calling the Rest endpoint for provisioning is no longer available.

Connections are defined in the provisioning.yaml file. The Agent will watch the file and resolve the desired state, applying connections defined in the file.

Defining a Connection

Connections are defined in the provisioning.yaml. This file is divided into components, each component representing a type of connection.

Each component is mandatory:

Name - This is the free name of the connection
Version set to 1
Configuration - This is a list of keys/values dependent on the component type.

Example provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm, the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and resolve the value at runtime.

sslKeystorePassword:
  value: ${ENV_VAR_NAME}

Referencing files

Many connections need files, for example, to secure Kafka with SSL you will need a key store and optionally a trust store.

To reference a file in the provisioning.yaml, for example, given:

    configuration:
      protocol:
        value: SASL_SSL
      sslKeystore:
        file: "my-keystore.jks"

a file called my-keystore.jks is expected in the same directory.

HQ

This page describes connection a Lenses Agent with HQ

To be able to view and drill into your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: hq-tls-test-lenses-hq.hq-agent-test.svc.cluster.local
      port:
        value: 10000
      agentKey:
        value: ${LENSESHQ_AGENT_KEY}
      sslEnabled:
        value: true
      sslTruststore:
        file: "/mnt/provision-secrets/grpc/truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

Kafka

This page describes how to connect the Lenses Agent to your Kafka brokers.

The Lenses Agent can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Apache Kafka

Connect the Lenses Agent to your Apache Kafka cluster.

Aiven

Connect the Lenses Agent to your Aiven Kafka cluster.

AWS MSK

Connect the Lenses Agent to your AWS MSK cluster.

AWS MSK Serverless

Connect the Lenses Agent to your AWS MSK Serverless.

Azure Event Hubs

Connect the Lenses Agent to your Azure Event Hubs.

Azure HDInsight

Connect the Lenses Agent to your Azure HDInsight cluster.

Confluent Cloud

Connect the Lenses Agent to your Confluent Cloud.

Confluent Platform

Connect the Lenses Agent to your Confluent Platform (on premise) cluster.

IBM Event Streams

Connect the Lenses Agent to your IBM Event Streams cluster.

Apache Kafka

This page describes connecting the Lenses Agent to Apache Kafka.

A Kafka connection is required for the agent to start. You can connect to Kafka via:

Plaintext (no credentials an unencrypted)
SSL (no credentials an encrypted)
SASL Plaintext and SASL SSL

Plaintext

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.
protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of the Agent does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword

SASL Plaintext vs SASL SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is:

The transport layer is encyrpted (SSL)
The SASL mechanism for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

To use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

When encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

SASL SSL

Mechanism PLAIN

Encrypted communication and basic username and password for authentication.

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";

Mechanism GSSAPI

In order to use Kerberos authentication, a Kerberos Connection should be created beforehand.

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword  
    saslMechanism: 
      value: GSSAPI
    saslJaasConfig:
      value: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";      
     keytab:
       file: /path/to/keytab.jks

SASL Plaintext

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

Mechanism SCRAM-SHA-256

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

Mechanism SCRAM-SHA-512

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-512
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

Aiven

This page describes configuring Lenses to connect to Aiven.

Find your Service URI

From the Aiven, locate your Service URI and set it as the bootstrap servers.

Configure Provisioning

Set the following in the provisioning.yaml, replacing Service URI, username and password from your Aiven account.

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[Service URI]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="[your-username]"
        password="[your-password]";

AWS MSK

This page describes connection the Lenses Agent to a AWS MSK cluster.

It is recommended to install the Agent on an EC2 instance or with EKS in the same VPC as your MSK cluster. The Agent can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Agent installation.

Enable Open Monitoring

If you want to have the Agent collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

It is not recommended to use Plaintext for secure environments. For these environments use TLS or IAM.

When the Agent is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

Configure Provisioning

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

AWS MSK Serverless

This page describes how to connect Lenses to an Amazon MSK Serverless cluster.

It is recommended to install the Agent on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster.

Security Groups

Enable communications between the Agent & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Agent installation.

IAM Policy

To authenticate the Agent & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Agent service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:topic/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:group/[cluster_name]/[cluster_uuid]/*"
        }
    ]
}

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Configure Provisioning

provisioning.yaml

kafka:
  tags: ["optional-tag"]
  name: kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandle

SQL Processor Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Topic*",
    "kafka-cluster:WriteData",
    "kafka-cluster:ReadData"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Group*"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

{
  "Action": [
    "glue:DeregisterDataPreview",
    "glue:ListRegistries",
    "glue:CreateRegistry",
    "glue:RegisterSchemaVersion",
    "glue:GetRegistry",
    "glue:UpdateRegistry",
    "glue:ListSchemas",
    "glue:DeleteRegistry",
    "glue:GetSchema",
    "glue:CreateSchema",
    "glue:ListSchemaVersions",
    "glue:GetSchemaVersion",
    "glue:UpdateSchema",
    "glue:DeleteSchemaVersions"
  ],
  "Resource": [
    "arn:aws:glue:[region]:[aws_account_id]:registry/*",
    "arn:aws:glue:[region]:[aws_account_id]:schema/*"
  ]
}

More details about how IAM works with MSK Serverless can be found in the documentation: MSK Serverless

Limitations

When using the Agent with MSK Serverless:

The agent does not receive Prometheus-compatible metrics from the brokers because they are not exported outside of CloudWatch.
The agent does not configure quotas and ACLs because MSK Serveless does not allow this.

Azure EventHubs

This page describes connection Lenses to Azure EventHubs.

Create a Data Integration API key

Add a shared access policy
- Navigate to your Event Hub resource and select Shared access policies in the Settings section.
- Select + Add shared access policy, give a name, and check all boxes for the permissions (Manage, Send, Listen)
Once the policy is created, obtain the Primary Connection String, by clicking the policy and copying the connection string. The connection string will be used as a JAAS password to connect to Kafka.
The bootstrap broker [YOUR_EVENT_HUBS_NAMESPACE].servicebus.windows.net:9093

Configure Provisioning

Set the following in the provisioning.yaml

First set environment variable

Note that "\" at "$ConnectionString" is set additionally to escape the $ sign.

terminal

export SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="\$ConnectionString" password="Endpoint=sb://[SB_URL]/;SharedAccessKeyName=[KEY_NAME];SharedAccessKey=[ACCESS_KEY]";

provision.yaml

kafka:
- name: kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
    saslJaasConfig:
      value: '${SASL_JAAS_CONFIG}'
    saslMechanism:
      value: PLAIN
    protocol:
      value: SASL_SSL

provision.yaml

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: [my-tag]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
          - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
      saslJaasConfig:
        value: org.apache.kafka.common.security.plain.PlainLoginModule required username="\$ConnectionString" password="Endpoint=sb://[SB_URL]/;SharedAccessKeyName=[KEY_NAME];SharedAccessKey=[ACCESS_KEY]";
      saslMechanism:
        value: PLAIN
      protocol:
        value: SASL_SSL

Azure HDInsight

This page describes connection Lenses to a Azure HDInsight cluster.

Find your Kafka endpoints

In our Azure Portal, go to Dashboards->Ambari home. Go to Kafka->Configs->Kafka Broker->Kafka Broker hosts

Optionally find your Zookeeper endpoints

Optionally get the Zookeeper endpoints: Go to Zookeeper->Configs->Zookeeper Server->Zookeeper Server hosts.

Configure Provisioning

See Apache Kafka for different configurations for your security protocols.

kafka:
- name: kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

Confluent Cloud

This page describes configuring Lenses to connect to Confluent Cloud.

For Confluent Platform see Apache Kafka.

Create a Data Integration API key

From Data integration API keys, select Create Key.
For this guide select Global access to get your API Key and API Secret Key.
Go to Cluster Settings to get your Bootstrap Server.

Configure Provisioning

Set the following in the provisioning.yaml

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required 
        username="[YOUR_API_KEY]" 
        password="[YOUR_API_KEY_SECRET]";

Confluent Platform

This page describes configuring Lenses to connect to Confluent Platform.

For Confluent Platform see Apache Kafka.

IBM Event Streams

This page describes how to connect Lenses to IBM Event Streams.

IBM Event Streams requires a replication factor of 3. Ensure you set the replication factor accordingly for Lenses internal topics.

See Configuration.

Find your bootstrap endpoints

From the IBM Cloud console, locate your bootstrap_endpoints, for the service credentials you want to connect with.

Configure Provisioning

Set the following in the provisioning.yaml:

Use "token" as the username in the Jaas Config. Set the password as your API KEY from IBM Event streams

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_ENDPOINTS]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required 
        username="token" 
        password="[YOUR_API_KEY]";

Schema Registries

This page describes adding a Schema Registries to the Lenses Agent.

Overview

Learn an overview of connecting the Lenses Agent to Schema Registries.

AWS Glue

Connect the Lenses Agent to your AWS Glue service for schema registry support.

Confluent

Connect the Lenses Agent to Confluent Schema Registry.

IBM Event Streams

Connect the Lenses Agent to IBM Event Streams Schema Registry

Apicurio

Connect the Lenses Agent to Apicurio.

Overview

This page describes an overview of connecting a Lenses Agent with Schema Registries

Consider Rate Limiting if you have a high number of schemas.

Authentication

TLS and basic authentication are supported for connections to Schema Registries.

JMX Metrics

The Agent can collect Schema registry metrics via:

JMX
Jolokia

Supported formats

AVRO
PROTOBUF

JSON and XML formats are supported by Lenses but without a backing schema registry.

Schema deletion

To enable the deletion of schemas in the UI, set the following in the lenses.conf file.

lenses.conf

## Enable schema deletion in the Lenses UI
## default: false
lenses.schema.registry.delete = true

## When a topic is deleted,
## automatically delete also its associated Schema Registry subjects
## default: false
lenses.schema.registry.cascade.delete = true

IBM Event Streams supports hard deletes only

AWS Glue

This page describes connection to AWS Glue.

AWS Glue Schema Registry connection, depends on an AWS connection.

Set the following examples in provisioning.yaml

These are examples of provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it.

Using AWS Access Key

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Access Key
      accessKeyId: 
        value: my-access-key-id
      secretAccessKey: 
        value: my-secret-access-key
      region: 
        value: eu-west-1
      
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        reference: my-aws-connection
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
          value: arn:aws:glue:region:123123123:registry/my-registry

Using AWS Credentials Chain

aws:
  - name: my-aws-connection
    version: 1
    tags: []
    configuration:
      region:
        value: eu-north-1
      authMode:
        value: "Credentials Chain"

glueSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: []
    templateName: SchemaRegistry
    configuration:
      authMode:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:eu-north-1:[account-id]:registry/[name]

Confluent

This page describes connecting Lenses to Confluent schema registries.

Set the following examples in provisioning.yaml

Simple configuration, with JMX metrics

The URLs (nodes) should always have a scheme defined (http:// or https://).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      ## all metrics properties are optional
      metricsPort: 
        value: 9581
      metricsType: 
        value: JMX
      metricsSsl: 
        value: false

Basic authentication

For Basic Authentication, define username and password properties.

confluentSchemaRegistry:
- name: schema-registry
  tags: ["tag1"]
  version: 1    
  configuration:
    schemaRegistryUrls:
      value:
        - http://my-sr.host1:8081
        - http://my-sr.host2:8081
    username: 
      value: my-username
    password: 
      value: my-password

TLS with custom truststore

A custom truststore is needed when the Schema Registry is served over TLS (encryption-in-transit) and the Registry’s certificate is not signed by a trusted CA.

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslKeystore:
        fileRef:
          filePath: /path/to/my/keystore.jks
      sslKeystorePassword: 
        value: myPassword

Hard or soft delete

By default, Lenses will use hard delete for Schema Registry. To use soft delete, add the following property:

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      hardDelete:
        value: true

Apicurio

This page describes connecting Lenses to Apicurio.

Apicuro supports the following versions of Confluent's API:

Confluent Schema Registry API v6
Confluent Schema Registry API v7

Set the following examples in provisioning.yaml

Set the schema registry URLs to include the compatibility endpoints, for example:

http://localhost:8080/apis/ccompat/v6

provisioning.yaml

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://localhost:8080/apis/ccompat/v6

IBM Event Streams Registry

This page describes connecting Lenses to IBM Event Streams schema registry.

Requires Enterprise subscription on IBM Event Streams and only hard delete is supported for IBM Event streams

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

https://token:{$APIKEY}@{$HOST}/confluent

Use "token" as the username. Set the password as your API KEY from IBM Event streams

Set the following examples in provisioning.yaml

provisioning.yaml

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - https://token:{$APIKEY}@{$HOST}/confluent

Kafka Connect

This page describes adding a Kafka Connect Cluster to the Lenses Agent.

Lenses integrates with Kafka Connect Clusters to manage connectors.

The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Multiple Kafka Connect clusters are supported.

If you are using Kafka Connect < 2.6 set the following to ensure you can see Connectors

lenses.features.connectors.topics.via.api.enabled=false

Consider Rate Limiting if you have a high number of connectors.

Simple configuration, with JMX metrics

The URLs (workers) should always have a scheme defined (http:// or https://).

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    version: 1    
    tags: ["tag1"]
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083
      metricsPort: 
        value: 9585
      metricsType: 
        value: JMX

Basic authentication

For Basic Authentication, define username and password properties.

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      username: 
        value: my-username
      password: 
        value: my-password

TLS with custom truststore

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      sslTruststore:
        file: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

provisioning.yaml

connect:
  name: my-connect-cluster-name
  tags: ["tag1"]
  version: 1    
  configuration:
    workers:
      value:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083    
    sslKeystore:
      file: /path/to/my/keystore.jks
    sslKeystorePassword: 
      value: myPassword

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

lenses.conf

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]

Zookeeper

This page describes adding a Zookeeper to the Lenses Agent.

Set the following examples in provisioning.yaml

Simple configuration, without metrics

zookeeper:
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds

Simple configuration, with JMX metrics

Simple configuration with Zookeeper metrics read via JMX.

zookeeper:    
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false

With such a configuration, Lenses will use 3 Zookeeper nodes and will try to read their metrics from following URLs (notice the same port - 9581 - used for all of them, as defined by metricsPort property):

my-zookeeper-host-0:9581
my-zookeeper-host-1:9581
my-zookeeper-host-2:9581

AWS

Add a connection to AWS in the Lenses Agent.

The agent uses an AWS in three places:

AWS IAM connection to MSK for Lenses itself
Connecting to AWS Glue
Alert channels to Cloud Watch.

If the Agent is deployed on an EC2 Instance or has access to AWS credentials in the default AWS toolchain that can be used instead.

provisioning.yaml

aws:
- name: my-aws-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # Way to authenticate against AWS.Credentials Chain or Access Key
    authMode:
      value:
    # Access key ID of an AWS IAM account.
    accessKeyId:
       value:
    # Secret access key of an AWS IAM account.
    secretAccessKey:
       value:
    # AWS region to connect to. If not provided, this is deferred to client 
    # configuration.
    region:
       value:
    # Specifies the session token value that is required if you are using temporary 
    # security credentials that you retrieved directly from AWS STS operations.
    sessionToken:
       value:

Alert & Audit integrations

Connect the Lenses Agent to your alerting and auditing systems.

The Agent can send out alerts and audits events. Once you have configured alert and audit connections, you can create alert and audit channels to route events to them.

Alerts

DataDog

provisioning.yaml

datadog:
- name: my-datadog-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The Datadog site.
    site:
      value:
    # The Datadog API key.
    apiKey:
      value:   
    # The Datadog application key.
    applicationKey:
      value:

AWS CloudWatch

See AWS connection.

PagerDuty

provisioning.yaml

pagerduty:
- name: my-pagerduty-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # An Integration Key for PagerDuty's service with Events API v2 integration type.
    integrationKey:
      value:

Slack

provisioning.yaml

slack:
- name: my-slack-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The Slack endpoint to send the alert to.
    webhookUrl:
      value:

Alert Manager

provisioning.yaml

alertManager:
- name: my-alertmanager-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # Comma separated list of Alert Manager endpoints.
    endpoints:
      value:

Webook (Email, SMS, HTTP and MS Teams)

provisioning.yaml

webhook:
- name: my-webhook-alert-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The host name for the HTTP Event Collector API of the Splunk instance.
    host:
      value: 
    # The port number for the HTTP Event Collector API of the Splunk instance. (int)
    port:
      value:  
    # Set to true in order to set the URL scheme to https. 
    # Will otherwise default to http.
    useHttps:
      value:
    # An array of (secret) strings to be passed over to alert channel plugins.
    creds:
      value:
        - 
        -

Audits

Webhook

provisioning.yaml

webhook:
- name: my-webhook-audit-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The host name for the HTTP Event Collector API of the Splunk instance.
    host:
      value: 
    # The port number for the HTTP Event Collector API of the Splunk instance. (int)
    port:
      value:  
    # Set to true in order to set the URL scheme to https. 
    # Will otherwise default to http.
    useHttps:
      value:
    # An array of (secret) strings to be passed over to alert channel plugins.
    creds:
      value:
        - 
        -

Splunk

provisioning.yaml

splunk:
- name: my-splunk-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The host name for the HTTP Event Collector API of the Splunk instance.
    host:
      value: 
    # The port number for the HTTP Event Collector API of the Splunk instance. (int)
    port:
      value:  
    # Use TLS. Boolean, default false
    useHttps:
      value:
    # This is not encouraged but is required for a Splunk Cloud Trial instance. Bool
    insecure:
      value:
    # HTTP event collector authorization token. (string)
    token:
      value:

Infrastructure JMX Metrics

This page describes how to configure JMX metrics for Connections in Lenses.

All core services (Kafka, Schema Registry, Kafka Connect, Zookeeper) use the same set of properties for services’ monitoring.

The Agent will discover all the brokers by itself and will try to fetch metrics using metricsPort, metricsCustomUrlMappings and other properties (if specified).

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: 
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: 
       value: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

For JOLOKIA each entry value in metricsCustomUrlMappings must contain protocol.

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

Before enabling collection of metrics within Agents provision configuration, make sure in your MSK Provisioned cluster you have enabled open monitoring with Prometheus.

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

kafka:
  tags: [ "dev", "dev-2", "eu"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://my-broker-0:9098
        - SASL_SSL://my-broker-1:9098
        - SASL_SSL://my-broker-2:9098
    protocol:
      value: SASL_SSL
    saslMechanism:
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsType:
      value: AWS

Custom URL mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

my-kafka-host-0:9582 - because of metricsCustomUrlMappings
my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol:
      value: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582