Lenses is the leading developer experience and UI for exploring and moving real-time data, across any Kafka, on any cloud or on premise. We are on a mission to create an operating fabric to increase developer productivity on real-time data.

Architecture

Lenses is an application that connects to your Kafka environments, allowing you to manage, discover, explore and catalogue your data via SQL. You can also deploy and monitor stream processing applications (SQL Processors) and Kafka Connectors, all wrapped in an enterprize grade RBAC layer.

All Lenses needs is connectivity to your services, think of it as a Kafka client.

The diagram gives a high-level overview of the logical components. At the core of Lenses, we have:

• A Kafka UI for day-to-day work with Kafka

• SQL Engine to query data and create streaming apps leveraging Kafka Streams

• App Engine to manage seamless deployments of SQL apps (deployed to Kubernetes)

• Metadata Engine to create a real-time Data Catalog for cross-system datasets and apps

Lenses is a JVM application exposes secure restful APIs and websockets in addition to providing a Kafka UI. A CLI is available to help automate operations.

Kafka versions

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud.

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

JMX connectivity

Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and OpenMetrics (MSK).

For more enable JMX for Lenses itself see here.

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 6GB RAM/4 CPUs and 500MB disk space.

Memory & CPU

This is the default configuration = Request 1 CPU & Memory 3Gi, Limit 2 CPU & Memory 5Gi

Browser

All recent versions of major browsers are fully supported.

APIs and Websockets

Every action in Lenses is backed by an API or websocket, documented at https://api.lenses.io. A Golang client is available and CLI (command line interface).

Lenses state store

Lenses can use an embedded H2 database or a Postgres database. Postgres is not supplied by Lenses.

TLS termination

By default, Lenses does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Lenses or by using a TLS proxy or load balancer. Refer to the TLS documentation for additional information.

1. Add your bootstrap brokers including ports

2. Optionally, security protocol e.g. SASL_SCRAM, SASL_SSL

3. Optionally, SASL Mechanism, e.g. SCRAM-SHA-256

SSL/TLS Configuration

If your Kafka connection requires TLS, set the following

• Truststore: The SSL/TLS trust store to use as the global JVM trust store. Available formats are .jks, .p12, .pfx.

• Keystore: The SSL/TLS keystore to use for the TLS listener for Lenses. Available format is .jks.

JMX Metrics

Lenses allows you to connect to brokers JMX. Supported formats are:

1. Simple with and without SSL

2. Jolokia (JOLOKIAG and JOLOKIAP)

   1. With and without SSL

   2. With Basic Auth

   3. Custom http requests and suffix

3. AWS Open Monitoring

For versions 4.0 to 5.4 see our legacy documentation.

5.5.10

Improvements

• Audit Logs include the list of groups presented by IdP for the user

• The group listing API endpoint now includes a list of users

5.5.9

Improvements

• Optimise kubernetes event handling

• Add extra logging for queue processing and event handling

5.5.8

Improvements

• Optimise topic auto-detection audit logging to avoid duplicate entries

• Optimise logging (adjust UDAF for intellisense polluting the logs, better actor mailbox logging)

• Improvements to the connector verification logic when Lenses has to mock topics or topics.regex

5.5.7

Improvements

This version improves the fetching of schemas from Schema Registries. The related subsystem has been re-worked to provide better error handling, fewer requests to the Schema Registry, and support rate-limiting. Find out how to configure rate limiting.

5.5.6

Improvements

• The S3 backup/restore functionality now supports the latest version of the Stream Reactor S3 connector plugin.

• New users coming from LDAP will not be created unless they have groups coming from LDAP matching Lenses groups. Users can still be created manually by an administrator.

5.5.5

Improvements

• Improve performance of the data catalogue. Lenses should now be many times faster to detect topics and their serialization, and use less memory and CPU time. For teams with Kafka clusters that have thousands of schemas, the startup time will also improve. For teams with tens of thousands of schemas, consumers, and partitions, software stability will also improve.

• Bring back the restart task button for paused connectors. This undocumented behaviour of Kafka Connect allows users to stop a connector’s consumer group, so they can reset offsets. For Kafka Connect 3.5 or later the new STOP connector API and corresponding button in Lenses can have the same effect.

• Compress schemas before sending them to the Schema Registry. This allows to send larger schemas to the Schema Registry as the limit is on the size of the request rather than the schema itself.

• Improvements to the Skip Validation option for inserting JSON messages, to allow for less strict (but still valid) schemas for inserted messages.

5.5.4

New Features

• Add STOP operation (button) for Connectors. The STOP operation requires Kafka Connect 3.5 or greater

• Allow to skip schema validation when inserting to JSON topics

Improvements

• Connector search is now case-insensitive

• Allow to type to search groups when creating service accounts

• Show masked passwords when editing a connector (regression in 5.5.3)

Fixes

• Filtering connectors by type doesn’t work

• When there were at least two Connect clusters with at least one connector with common name in both clusters, filtering connectors returns incorrect or multiple results

• Validating connectors with passwords may not work (regression in 5.5.3)

5.5.3

New Features

Support for case-insensitive LDAP users

Whilst Lenses users are case-sensitive, LDAP most of the time performs case-insensitive searches on user accounts. This can lead to users who try to login to Lenses with different casing in their username (e.g., user and USER) to get duplicate accounts.

We added the option lenses.security.ldap.case.sensitive with a default value of true. It can be switched to false in which case Lenses will treat usernames from LDAP as case-insensitive and always converting to lowercase.

Improvements

• Upgrade the AWS IAM library to better support service account roles inside EKS

• Upgrade libraries with known CVEs —not affecting Lenses in either way

Fixes

• Fix Grafana link not showing up on sidebar

• Fix a case where some sensitive data might leak in the logs

• Fix filtering by connector name causing the connector screen to crash if a connect cluster is offline

5.5.2

Improvements

• The connectors’ screen will not mask passwords if they are referencing a secret from a secret provider.

Fixes

• Fix regression where connectors’ passwords were not masked.

5.5.1

Improvements

• Authentication:

  • Enhanced authentication to reject with a 401 status code when the user lacks any attached groups in the IdP (Identity Provider).

  • Improved authentication flow, allowing an authenticated SSO (Single Sign-On) user to log in even if there isn’t a corresponding group in Lenses.

• Documentation Enhancement:

  • New SQL processor page with direct links to the latest documentation and support resources for user convenience.

Fixes

• Deployment Issue:

  • Addressed a bug introduced in Lenses git ops deployment* version 5.5.0, resolving provisioning issues experienced in certain deployment scenarios.

• SSO Authentication Fix:

  • Corrected SSO authentication behavior. When an SSO user is configured to overwrite the IdP groups, Lenses now correctly refrains from extracting groups from the IdP.

5.5.0

New Features

Kafka Connectors as Code

Lenses now introduces support for managing Kafka connectors as code. With this feature, you can define your connectors in a YAML file and seamlessly deploy them to Lenses. This capability is accessible via both the Lenses CLI and the Lenses UI. This release marks the commencement of our journey towards a more declarative and automated approach to managing Kafka and Lenses resources.

Consumer Group Management

In this version, Lenses introduces support for deleting consumer group offsets and entire consumer groups, enhancing flexibility and control over consumer group management.

Generic SSO Provider

Lenses provides support for a few SSO providers out of the box like Google, Okta, etc. In this release, Lenses introduces a generic SSO provider, enabling users to integrate with any SSO provider that supports the SAML 2.0 protocol. This feature is configurable via the lenses.conf file under lenses.security.saml.idp.provider.

Enhancements

Kafka Message Replay

The Kafka message replay feature receives an enhancement, now enabling users to replay messages from a specific offset. This functionality is accessible from both the Lenses topic screen and the Lenses SQL studio screen, providing greater precision in message replay operations.

Consumer Group Offsets Data Link

Users can now seamlessly navigate from the consumer group offsets screen to the data of the topic that the consumer group offset points to, enhancing visibility and ease of data exploration.

Audits to log file

Lenses now provides the capability to log audit events to its log file, enabling users to store audit logs locally for compliance and security purposes. This feature is configurable via the lenses.conf file under lenses.audit.to.log.file.

Lenses Internal Topics Replication Factor

To ensure compatibility with cloud providers such as IBM, where a minimum replication factor is mandated, Lenses now allows the configuration of the replication factor for its internal topics. This setting can be configured in the lenses.conf file under lenses.internal.topics.replication.***.

Bug Fixes

External Applications via Lenses SDK

The Lenses SDK, a thin client facilitating the monitoring and tracking of external applications connected to Kafka within Lenses topology, has been enhanced in this release. An issue where the application’s status in Lenses was not updated correctly has been resolved.

S3 Backup-Restore for JSON Payloads

In this release, a bug affecting the S3 backup-restore feature for JSON payloads has been rectified. Previously, the feature encountered issues due to the Connect converter enforcing schema on JSON payloads, leading to incorrect functionality. This bug has been addressed to ensure seamless backup and restoration of JSON data via S3.

Ugrade Notes

Lenses 5.5 is an incremental release which brings in new features and improvements.

Upgrading from 5.0 or later does not require any static configuration change but if you have automated the creation of any AWS connection, then you will have to adjust the provisioning section of your Helm chart, or your CICD, or —if you use the API directly— your API calls.

If you are upgrading from version 4.3 or older, you need to follow the upgrade procedure for Lenses 5.0 as well as the rest of the instructions that follow.

Breaking Changes and Caution Items

Lenses upgrades (except patch releases) are not backwards compatible. It is best practice to take a backup of the Lenses database before an upgrade.

New provisioning API [caution]

With Lenses 5.3 the provisioning API was introduced. This new API can be used to create or update the connections landscape. The old provisioning methods could only create the connection landscape (first run).

What this means, is that now the Helm chart or a CICD process can be used to manage Lenses’ connections.

For teams that are on the old provisioning method some adjustments are required to their Helm charts or other provisioning code to switch to the new API. The old methods are still available but are considered deprecated and will be removed or break in the future.

AWS and Glue Connection provisioning [breaking]

With Lenses 5.4 IAM support was added for the AWS connection type. An AWS connection is used as an authentication provider for the Glue Schema Registry and Cloudwatch channels.

Due to this change, if you create or manage your AWS and Glue connections via the API or provisioning, you need to update your configuration to the new format.

Action required

• Add the new authMode property to your connections for AWS and Glue Schema Registry.

Details

• Lenses 5.4 adds a new required property for the AWS and Glue Schema Registry connections.

• The property is authMode.

• It controls how Lenses authenticates with AWS:

  • Access keys (existing feature).

  • Credentials provider chain (new feature).

• You set the property either with the:

  • Connections API - create, update.

  • Provision YAML.

You can set authMode in 2 modes:

1. Access keys mode

This is the existing mode where Lenses uses AWS access keys.

• Set the authMode to Access Key.

• Specify the access key ID and secret access key, as you had before.

{
  "authMode": { "value": "Access Key" },
  "accessKeyId": { "value": "yourAccessKeyId" },
  "secretAccessKey": { "value": "yourSecretAccessKey" }
}

2. Credentials provider chain mode (new)

This is the new mode where Lenses uses the AWS default credentials provider chain.

• Set the authMode to Credentials Chain.

• No additional properties needed.

{
  "authMode": { "value": "Credentials Chain" }
}

Examples - Provision YAML

1. Access mode

aws:
  - name: my-aws-connection
    version: 1
    tags: [dev]
    configuration:
      authMode:
        value: Access Key
      accessKeyId:
        value: yourAccessKeyId
      secretAccessKey:
        value: yourSecretAccessKey

glueSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: [dev]
    configuration:
      authMode:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:account:registry/registry-name
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection

2. Credentials provider chain mode

aws:
  - name: my-aws-connection
    version: 1
    tags: [dev]
    configuration:
      authMode: 
        value: Credentials Chain

glueSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: [dev]
    configuration:
      authMode:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:account:registry/registry-name

Examples - API JSON

1. Access mode

AWS connection

{
 "name": "my-aws-connection",
 "tags": ["dev"],
 "templateName": "AWS",
 "configuration": {
   "authMode": { "value": "Access Key" },
   "accessKeyId": { "value": "yourAccessKeyId" },
   "secretAccessKey": { "value": "yourSecretAccessKey" }
 }
}

Glue Schema Registry connection

{
    "name":"schema-registry",
    "tags": ["dev"],
    "templateName":"AWSGlueSchemaRegistry",
    "configuration": {
        "authMode": {"reference":"my-aws-connection"},
        "accessKeyId": {"reference":"my-aws-connection"},
        "secretAccessKey": {"reference":"my-aws-connection"},
        "glueRegistryArn":{"value":"arn:aws:glue:region:account:registry/registry-name"}
    }
}

2. Credentials provider chain mode

AWS connection

{
  "name": "my-aws-connection",
  "tags": ["dev"],
  "templateName": "AWS",
  "configuration": {
    "authMode": { "value": "Credentials Chain" }
  }
}

Glue Schema Registry connection

{
    "name":"schema-registry",
    "tags": ["dev"],
    "templateName":"AWSGlueSchemaRegistry",
    "configuration": {
        "authMode": {"reference":"my-aws-connection"},
        "glueRegistryArn":{"value":"arn:aws:glue:region:account:registry/registry-name"}
    }
}

Docker image base change

Starting with Lenses 5.2 the base image of Lenses and SQL Processor Dockers switched from Debian to Ubuntu. On some older systems, these docker images will fail to run, due to a combination of a recent glibc in the container, and older docker daemon on the host.

If you fall under this category, during the startup of the Lenses container, you might see errors such as Unable to identify system. Uname is required or [warning][os,thread] Failed to start thread “GC Thread#0”.

For these cases, we now offer Lenses docker images with the suffix -debian in their tags. E.g:

• lensesio/lenses:5.5-debian

• lensesio/lenses:5.5.0-debian

• lensesio/lenses:latest-debian

If your host is running on an older operating system and you encounter these errors, try to use the debian equivalent tag.

Update Process

Using the Lenses Archive

Download the latest 5.5 archive and extract it in a new directory on your server. It is important to avoid extracting an archive over an older installation to avoid having multiple versions of libraries. Instead, you should remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files. If you are using the internal database instead of PostgreSQL, make sure Lenses Storage Directory (lenses.storage.directory) is kept intact. The folder is where persistent data is stored, such as users, groups, audits, data policies, connections, and more.

Make sure you have a JRE (or JDK) installed in the server running Lenses. Lenses can run on JRE 8 or greater, and the recommended version is JRE 11.

Using the Lenses Docker

The docker image uses tags to distinguish between versions. The latest tag (lensesio/lenses:latest) brings the latest stable version of Lenses. There are minor tags to help users get the latest patch in a minor version (e.g 5.5, 5.1) and patch tags to help users pin to a specific patch (e.g 5.5.1, 5.1.2). The best practice advice is to use the minor tag (lensesio/lenses:5.5), which ensures that your installation will always get compatible updates until you made a conscious decision to upgrade the minor version.

If you use the internal database instead of PostgreSQL as the backing store of Lenses, make sure you keep the /data/storage volume to not lose your data. Other volumes supported by the docker are /data/kafka-streams-state which holds state for SQL Processors running IN-PROC and may have to be rebuilt (automatically) if lost, /data/log (log files on disk), /data/plugins (custom UDFs).

Pull the 5.5 docker:

docker pull lensesio/lenses:5.5

Stop your current container and restart with the 5.5 image, mounting any volumes you might need.

Lenses Box

If you are a Box user, pull the latest version, preserve your /data volume and restart Lenses:

docker pull lensesio/box:5.5
docker stop [CURRENT BOX NAME or ID]
docker run -v /path/to/box/data:/data -e EULA="..." -p 3030 lensesio/box:5.5

Helm

Download the latest charts and update your values.yaml as described below. Remember that if you are using the internal database instead of PostgreSQL as the backing store, then the Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates. To support a potential downgrade, make sure this volume is backed-up before installing a newer version of Lenses.

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

If you have provisioning enabled (lenses.provision.enabled: true) in your values.yaml, and you are on provision version “1” then you have to act. Version “1” means either that lenses.provision.version is set to "1", or it is not set at all. You have two options:

• Disable it, as Lenses already has all the information stored in the database, and version “1” does not support updating the connections and license.Copy

  Copy

  lenses:
    provision:
      enabled: false

• Switch to provisioning version “2” which supports updating connections and licenses every time you do a helm upgrade. To do that, you must make some changes to your old provisioning section. Some resources that can come handy for the switch are:

  • Provisioning API introduction

  • Provisioning API OpenAPI reference

  • Helm chart examples

If you don’t have your values.yaml you can download it from the Kubernetes cluster using Helm:

helm get values --namespace [LENSES_NAMESPACE] \
     --output yaml [LENSES_DEPLOYMENT] > values.yaml

Proceed to upgrade:

helm upgrade --namespace [LENSES_NAMESPACE] --values values.yaml [LENSES_DEPLOYMENT]

Alternatively, reusing the old values and turning provisioning off:

helm upgrade --namespace [LENSES_NAMESPACE] --reuse-values \
     --set lenses.provision.enabled=false [LENSES_DEPLOYMENT]

Cloud Installations

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates. If a new image does not exist, you may be able to update Lenses in-place. Our support team will be happy to go through the available options with you.

\

Create a Data integration API key

1. From Data integration API keys, select Create Key.

2. For this guide select Global access

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol SASL SSL

2. SASL Mechanism PLAIN

In the JAAS Configuration update the username and password from the respective fields Key and Secret of the API key created above:

org.apache.kafka.common.security.plain.PlainLoginModule required 
username="[Your_API_KEY]" 
password="[Your_API_KEY_SECRET]"

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports

2. Enable basic auth if required and set the user name and password

3. Enable SSL if required and upload the keystore

4. Optionally upload a trust store

5. Set any additional properties

6. Optional enable metrics

Lenses integrates with Kafka Connect Clusters to manage connectors.

Adding a connection

To add a connection, go to Admin->Connections->New Connection->Kafka Connect.

1. Provide a name for the Connect cluster

2. Add a comma-separated list of the workers in the Connector cluster, including ports

3. Optionally enable Basic Auth and set the username and password

4. Optionally enable SSL and upload the key-store file

5. Optionally upload a trust store

6. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]

Lenses requires a valid license to start. The license can be added via the UI when in bootstrap mode or at deployment time via the provisioning APIs.

License Expired

If at any point the license becomes invalid (it expired / too many brokers were added to the cluster) only the license page will be available.

License Management

See License Management.

Apicuro supports the following versions of Confluent's API:

• Confluent Schema Registry API v6

• Confluent Schema Registry API v7

Set the schema registry URLs to include the compatibility endpoints, for example:

http://localhost:8080/apis/ccompat/v6

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports and compatibility endpoint path

2. Enable basic auth if required and set the user name and password

Connectivity to Zookeeper is optional for Lenses. Zookeeper is used by Lenses for such purposes:

1. To provide quotas management (until quotas can be managed via the Brokers API)

2. To autodetect the JMX connectivity settings to Kafka brokers (if metrics are not defined directly for Kafka connection).

To add a Zookeeper connection go to Admin->Connections->New Connection->Zookeeper.

1. Add a comma-separated list of Zookeepers, including port

2. Optionally set a session timeout

3. Optionally set a Chroot path

4. Optionally set a connection timeout

5. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

https://token:{$APIKEY}@{$HOST}/{confluent}

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports, adding the confluent path at the end. Use the value from the kafka_http_url field in the IBM Console Service Credentials tab

2. Enable basic auth if required

3. Set the user name "token"

4. Set the password as the value from API key in the IBM Console Service Credentials tab

Lenses can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Follow the guide for your distribution to obtain the credentials and bootstrap broker to provide to Lenses.

To connect Lenses to your real environment you can:

1. Install Lenses (not the Box) and manually configure the connections to Kafka, Zookeepers, Schema Registries and Connect, or

2. Install Lenses and configure the connections in one go using provisioning.

How to connect to Kafka depends on your Kafka provider.

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Edit the relevant Security Group

Enable communications between Lenses & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Lenses installation.

Configure IAM Policies

To authenticate Lenses & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Lenses service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:topic/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:group/[cluster_name]/[cluster_uuid]/*"
        }
    ]
}

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Creating the Connection in Lenses

In the Lenses bootstrap UI, Select:

1. For the bootsrap server configuration, use the MSK Serverless endpoint

2. For the Security Protocol, set it to SASL_SSL

3. Customize the Sasl Mechanism and set it to AWS_MSK_IAM

4. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

1. During the broker metrics export step, keep it disabled, as AWS Serverless does not export the metrics to Lenses. Click Next

2. Copy your license and add it to Lenses, validate your license, and click Next

3. Click on Save & Boot Lenses. Lenses will finish the setup on its own

Additional Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Topic*",
    "kafka-cluster:WriteData",
    "kafka-cluster:ReadData"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Group*"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

{
  "Action": [
    "glue:DeregisterDataPreview",
    "glue:ListRegistries",
    "glue:CreateRegistry",
    "glue:RegisterSchemaVersion",
    "glue:GetRegistry",
    "glue:UpdateRegistry",
    "glue:ListSchemas",
    "glue:DeleteRegistry",
    "glue:GetSchema",
    "glue:CreateSchema",
    "glue:ListSchemaVersions",
    "glue:GetSchemaVersion",
    "glue:UpdateSchema",
    "glue:DeleteSchemaVersions"
  ],
  "Resource": [
    "arn:aws:glue:[region]:[aws_account_id]:registry/*",
    "arn:aws:glue:[region]:[aws_account_id]:schema/*"
  ]
}

More details about how IAM works with MSK Serverless can be found in the documentation: MSK Serverless

Limitations

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Lenses installation.

Enable Open Monitoring

If you want to have Lenses collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol and set the protocol you want to use

2. SASL Mechanism and set the mechanism you want to use.

Connecting with AWS IAM

In the Lenses bootstrap UI, Select:

1. Security Protocol and set it to SASL_SSL

2. Sasl Mechanism and set it to AWS_MSK_IAM

3. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

4. Optionally upload your trust store

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

Lenses can work with the following schema registry implementations which can be added via the Connections page in Lenses.

Go to Admin->Connections->New Connections->Schema Registry and follow the guide for your registry provider.

Authentication

TLS and basic authentication are supported for connections to Schema Registries.

JMX Metrics

Lenses can collect Schema registry metrics via:

1. JMX

2. Jolokia

Supported formats

• AVRO

• PROTOBUF

To connect your Schema Registry with Lenses, select Schema Registry -> Create Connection.

Schema deletion

To enable the deletion of schemas in the UI, set the following in the lenses.conf file.

## Enable schema deletion in the Lenses UI
## default: false
lenses.schema.registry.delete = true

## When a topic is deleted,
## automatically delete also its associated Schema Registry subjects
## default: false
lenses.schema.registry.cascade.delete = true

Lenses can be deployed in the following ways:

To install Lenses from the archive you must:

1. Extract the archive

2. Configure Lenses

3. Start Lenses

Extracting the archive

Extract the archive using the following command

Inside the extract archive, you will find.

Starting Lenses

Start Lenses by running:

or pass the location of the config file:

If you do not pass the location of the config file, Lenses will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the security.conf to be readable only by the lenses user.

The agent needs write access in 4-5 places in total:

1. [RUNTIME DIRECTORY] When Lenses runs, it will create at least one directory under the directory it is run in:

   1. [RUNTIME DIRECTORY]/logs Where logs are stored

   2. [RUNTIME DIRECTORY]/logs/lenses-sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.

   3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.

   4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.

   5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

JNI libraries

Lenses and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

1. Mount /tmp without noexec

2. or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

SystemD example

If your server uses systemd as a Service Manager, then manage Lenses (start upon system boot, stop, restart). Below is a simple unit file that starts Lenses automatically on system boot.

Global Truststore

Lenses uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, Secure LDAP, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (e.g. Secure LDAP and JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

Hardware & OS

Run on any Linux server. For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Increase as a super-user the soft limit to 4096 with:

Use 6GB RAM/4 CPUs and 500MB disk space.

Lenses provides support for AWS Glue to manage schema and also explore and process data linked to it via the Lenses

To connect to Glue, first create a AWS Connection. Go to Admin->Connections->AWS and enter your AWS credentials or select the IAM support if Lenses is running on an AWS host, e.g. EC2 instance or it has the AWS default credentials toolchain provider in place.

Next, Select New Connection->Schema Registry->AWS Glue. Select your AWS Connection with access to Glue, and enter the Glue ARN.

Lenses can send out alerts and audits events, the following integrations are supported:

Alerts

1. DataDog

2. AWS CloudWatch

3. PagerDuty

4. Slack

5. Alert Manager

6. Webook (Email, SMS, HTTP and MS Teams)

Audits

1. Webhook

2. Splunk

Once you have configure alert and audit connections, you can create alert and audit channels to route events to them. See or for more information.

The Lenses docker image can be configured via environment variables or via volume mounts for the configuration files (lenses.conf, security.conf).

Running the Docker

Open Lenses in your , log in with admin/admin and configure your and add your .

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses.conf and security.conf can be mounted directly as

• /mnt/settings/lenses.conf

• /mnt/secrets/security.conf

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

• /data/storage

• /data/plugins

• /data/logs

• /data/kafka-streams-state

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Lenses upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Lenses may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Lenses TLS and Global JVM Trust Store

By default, the Lenses serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

1. Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets

2. Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting Lenses.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.

The AWS Marketplace offering requires AWS MSK (Managed Apache Kafka) to be available. Optionally, AWS RDS (or any other PostgreSQL-compatible database) can be configured for Lenses to store its state.

The following AWS resources are created:

• An EC2 instance that runs Lenses;

• A SecurityGroup to allow network access to the Lenses UI;

• A SecurityGroupIngress for Lenses to connect to MSK;

• A CloudWatch LogGroup where Lenses stores its logs;

• An IAM Role to allow the EC2 instance to store logs;

• An IAM InstanceProfile to pass the role to the EC2 instance;

• Optionally if enabled during deployment: an IAM Policy to allow the EC2 instance to emit CloudWatch metrics.

Deployment takes approximately three minutes.

AWS Marketplace Installation

Select CloudFormation Template, Lenses EC2 and your region.

Choose Launch CloudFormation.

Continue with the default options for creating the stack in the AWS wizard.

Fill in the parameters at Specify stack details.

• Deployment Here the EC2 instance size and password for the Lenses admin user are set. A t2.large instance size is recommended;

• Network Configuration This section controls the network settings of the Lenses EC2 instance. The ingress allows access to the Lenses UI only from particular IP addresses;

• MSK Set the Security Group ID to that of your MSK cluster. A rule will be added to it so that Lenses can communicate with your cluster. You can find the ID by navigating in the AWS console to your MSK cluster and then under Properties -> Networking settings;

• Monitoring Optionally produce the Lenses logs to CloudWatch;

• Storage Lenses stores its state in a database locally on the EC2 instance’s disk or in a PostgreSQL database. Local storage is a development/quickstart option and is not suitable for production use. It is advised to use a Postgres database for smoother upgrades.

Review the stack.

Accept the terms and conditions and create the stack.

Once the stack has deployed, go to the Output tab and click on the FQDN link. If there are no outputs listed you might need to press the refresh button.

Login to Lenses with admin and the password value you have submitted for the parameter LensesAdminPassword.

IAM Support

Lenses supports connection to MSK brokers via IAM. If Lenses is deployed on an EC2 instance it will use the default credential chain loader to authenticate and connect to MSK.

Supported Regions

The following Regions are supported:

• us-east-1;

• us-east-2;

• us-west-1;

• us-west-2;

• ca-central-1;

• eu-central-1;

• eu-west-1;

• eu-west-2;

• eu-west-3;

• ap-southeast-1;

• ap-southeast-2;

• ap-south-1;

• ap-northeast-1;

• ap-northeast-2;

• sa-east-1.

Security Recommendations

Please:

• Do not use your AWS root user for deployment or operations;

• Follow the least privileges principle when granting access to individual IAM user accounts;

• Avoid allowing traffic to the Lenses UI from a broad CIDR block where a more specific block could be used.

Pricing

AWS billing applies for the EC2 instance, CloudWatch logs and optionally CloudWatch metrics.

Troubleshooting

In case you run into problems, e.g. you cannot connect to Lenses, then the logs could provide more information. The easiest route to do this is to go to CloudWatch in the AWS console. Here, find the log group corresponding to your deployment (it has the same name as the deployment) and pick a log stream. The stream with the /lenses.log suffix contains all log lines regardless of the log level; the stream with the /lenses-warn.log suffix only contains warning-level logs.

If the above fails, for example, because the logs integration is broken, you can SSH into the EC2 instance. Lenses is installed into /opt/lenses, the logs can be found under /opt/lenses/logs for further inspection

In our Azure Portal, go to Dashboards > Ambari home.

1. Kafka endpoints: Go to Kafka > Configs > Kafka Broker > Kafka Broker hosts

2. Optionally get the Zookeeper endpoints: Go to Zookeeper > Configs > Zookeeper Server > Zookeeper Server hosts.

In the Lenses bootstrap UI:

1. Set the Kafka endpoints as bootstrap servers

2. Set the security protocol, mechanism and Jaas config according to your setup. For information on configuring clients (Lenses) for your HDInsight cluster see for unauthenticated and for authenticated.

TLS without Authentication

Set the following:

1. security.protocol to SSL

2. Set the password for your trust store

3. Upload your trust store

TLS with Authentication

Perform the additional steps to above

1. Set the password for your key store

2. Upload your key store

3. Set your key password

What’s in the Box?

Lenses Box contains all components of the Apache Kafka ecosystem, CLI tools, and synthetic data streams.

Starting the Box

2. Install and run the Docker

Kafka Docker advertisement

The broker in the Kafka docker has a broker id 101 and advertises the listener configuration endpoint to accept client connections.

If you run Docker on macOS or Windows, you may need to find the address of the VM running Docker and export it as the advertised listener address for the broker (On macOS it usually is 192.168.99.100). At the same time, you should give the lensesio/box image access to the VM’s network:

If you run on Linux, you don’t have to set the ADV_HOST , but you can do something cool with it. If you set it to be your machine’s IP address, you can access Kafka from any clients in your network.

If you decide to run a box in the cloud, you (and all your team) can access Kafka from your development machines. Remember to provide the public IP of your server as the kafka advertised host for your producers and consumers to access it.

Kafka Docker JMX

Kafka JMX metrics are enabled by default. Refer to ports once you expose the relevant port, ie. -p 9581:9581 you can connect to JMX with

Custom hostname

If you are using docker-machine or setting this up in a Cloud or DOCKER_HOST is a custom IP address such as 192.168.99.100, you will need to use the parameters --net=host -e ADV_HOST=192.168.99.100.

Docker data persistence

To persist the Kafka data between multiple executions, provide a name for your Docker instance and do not set the container to be removed automatically (--rm flag). For example:

Once you want to free up resources, just press Control-C. Now you have two options: either remove the Docker:

Or use it at a later time and continue from where you left off:

Port Numbers

Advanced options

FAQ

How can I run offline?

Download your key locally and run the command:

How much memory to allocate?

The container is running multiple services, and it is recommended to allocate 5GB of RAM to the docker (although it can operate with even less than 4GB).

To reduce the memory footprint, it is possible to disable some connectors and shrink the Kafka Connect heap size by applying these options (choose connectors to keep) to the docker run command:

Lenses uses an AWS in two places:

1. AWS IAM connection to for Lenses itself

3. to Cloud Watch.

First, add the Helm Chart repository using the Helm command line:

Use helm to install Lenses with default values:

The default install of Lenses will place Lenses in bootstrap mode, you can add the connections to Kafka manually and upload your license or automation with provisioning. Please refer to the GitHub values.yaml for all options.

Provisioning

To automatically provision the connections to Kafka and other systems set the .Values.lenses.provision.connections to be the YAML definition of your connections. For a full list of the connection types supported see .

The chart will render the full YAML specified under this setting as the provisioning.yaml file.

Alternatively you can use a second YAML file, which contains only the connections pass them at the command line when installing:

Helm Chart components

The chart uses:

1. Secrets to store Lenses Postgres credentials and authentication credentials

2. Secrets to store connection credentials such as Kafka SASL_SCRAM password or password for SSL JKS stores.

3. Secrets to hold the base64 encoded values of the JKS stores

4. ConfigMap for Lenses configuration overrides

5. Cluster roles and role bindings (optional).

Secrets and config maps are mounted as files under the mount /mnt:

1. settings - holds the lenses.conf

2. secrets - holds the secrets Lenses and license

3. provision-secrets - holds the secrets for connections in the provisioning.yaml file

4. provision-secrets/files - holds any file needed for a connection, e.g. JKS files.

Cluster RBAC

The Helm chart creates Cluster roles and bindings, these are used by SQL Processors if the deployment mode is set to KUBERENTES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.

To disable the RBAC set: rbacEnabled: false

If you want to limit the permissions Lenses has against your Kubernetes cluster, you can use Role/RoleBinging resources instead.

To achieve this you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to.

For example:

• Lenses namespace = lenses-ns

• Processor namespace = lenses-proc-ns

Finally you need to define in Lenses configuration which namespaces can Lenses access. To achieve this amend values.yaml to contain the following:

lenses.conf

The main configurable options for lenses.conf are available in the values.yaml under the lenses object. These include:

• Authentication

• Database connections

• SQL processor configurations

To apply other static configurations use lenses.append.conf, for example:

Authentication secrets

Set accordingly under**lenses.security.**

For SSO set lenses.security.saml

Postgres

To use Postgres as the backing store for Lenses set the details in the lenses.storage.postgres object.

If Postgres is not enabled a default embedded H2 database is used. To enable persistence for this data:

External Secrets

The chart relies on secrets for sensitive information such as Passwords. Secrets can rotate and are commonly stored in an external store such as Azure KeyVault, Hashicorp Vault or AWS Secrets Manager.

If you wish to have the chart use external secrets that are synchronized with these providers, set the following for the Lenses user:

For Postgres, add additional ENV variables via the lenses.additionalEnv object to point to your secret and set the username and password to external in the Postgres section.

Ingress & Services

Ingress and service resources are supported.

Enabled an Ingress resource in the values.yaml:

Enable a service resource in the values.yaml:

Controlling resources

To control the resources used by Lenses:

Enabling SQL Processors in K8s mode

To enable SQL processor in KUBERENTES mode and control the defaults:

Prometheus metrics

Prometheus metrics are automatically exposed on port 9102 under /metrics.

Example Values files

Building on the provisioning.yaml, API provisiong allows for uploading the files directly Lenses from anywhere with network access and without access to the host where Lenses is installed.

Uploading supporting files

Many connections need files, for example, to secure Kafka with SSL you will need a keystore and optionally a trust store.

To reference a file in the, for the configuration option set the key to be "file" and the value to reference in the API request. For example, given:

To upload the file to be used for the configuration option sslKeystore: add the following to the request:

API Call

1. Set the type to application/octet-stream.

2. The name of the part in the multipart request (supporting files) should match the value of the property pointing to the mounted file in the provisioning.yaml descriptor. This ensures accurate mapping and referencing of files.

3. Set LENSES_SESSION_TOKEN as the value of the Lenses Service Account token you want to use to automate provisioning.

In this example, the provisioning.yaml is read from provisioning=@"resources/provisioning.yaml.

The provisioning.yaml contains a reference to "my-keystore-file" which is loaded from @${PATH_TO_KEYSTORE_FILE};type=application/octet-stream

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime. i.e. inject an environment variable from GitHub secrets for passwords.

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582

Custom url mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

• my-kafka-host-0:9582 - because of metricsCustomUrlMappings

• my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

Simple configuration, without metrics

zookeeper:
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds

Simple configuration, with JMX metrics

Simple configuration with Zookeeper metrics read via JMX.

zookeeper:    
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false

With such a configuration, Lenses will use 3 Zookeeper nodes and will try to read their metrics from following URLs (notice the same port - 9581 - used for all of them, as defined by metricsPort property):

• my-zookeeper-host-0:9581

• my-zookeeper-host-1:9581

• my-zookeeper-host-2:9581

PLAINTEXT

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

• kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.

• protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL 

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword