Lenses is the leading developer experience and UI for exploring and moving real-time data, across any Kafka, on any cloud or on premise. We are on a mission to create an operating fabric to increase developer productivity on real-time data.

Architecture

Lenses is an application that connects to your Kafka environments, allowing you to manage, discover, explore and catalogue your data via SQL. You can also deploy and monitor stream processing applications (SQL Processors) and Kafka Connectors, all wrapped in an enterprize grade RBAC layer.

All Lenses needs is connectivity to your services, think of it as a Kafka client.

The diagram gives a high-level overview of the logical components. At the core of Lenses, we have:

• A Kafka UI for day-to-day work with Kafka

• SQL Engine to query data and create streaming apps leveraging Kafka Streams

• App Engine to manage seamless deployments of SQL apps (deployed to Kubernetes)

• Metadata Engine to create a real-time Data Catalog for cross-system datasets and apps

Lenses is a JVM application exposes secure restful APIs and websockets in addition to providing a Kafka UI. A CLI is available to help automate operations.

Changelog

For versions 4.0 to 5.4 see our .

5.5.14

New Features

• Add support for MSSQL as a backing store for Lenses.

Improvements

• LDAP connection management to avoid connection reset.

• Extra debug logging for when the Schema Registry sends an invalid Content-type header.

5.5.13

Fixes

• A security issue has been addressed.

5.5.12

Improvements

• The webhook for audits now offers the {{CONTENT}} variable to insert all the details of the audit log entry.

• Improve Kubernetes watchers and handling of SQL Processor Initialization events to avoid blocking operations.

5.5.11

Improvements

• The login audit now tracks both source IdP groups and applied groups.

5.5.10

Improvements

• Login audit now tracks source IdP groups.

• The Group Details API now includes user and service accounts within each group.

5.5.9

Improvements

• Optimise kubernetes event handling

• Add extra logging for queue processing and event handling

5.5.8

Improvements

• Optimise topic auto-detection audit logging to avoid duplicate entries

• Optimise logging (adjust UDAF for intellisense polluting the logs, better actor mailbox logging)

• Improvements to the connector verification logic when Lenses has to mock topics or topics.regex

5.5.7

Improvements

5.5.6

Improvements

• The S3 backup/restore functionality now supports the latest version of the Stream Reactor S3 connector plugin.

• New users coming from LDAP will not be created unless they have groups coming from LDAP matching Lenses groups. Users can still be created manually by an administrator.

5.5.5

Improvements

• Improve performance of the data catalogue. Lenses should now be many times faster to detect topics and their serialization, and use less memory and CPU time. For teams with Kafka clusters that have thousands of schemas, the startup time will also improve. For teams with tens of thousands of schemas, consumers, and partitions, software stability will also improve.

• Bring back the restart task button for paused connectors. This undocumented behaviour of Kafka Connect allows users to stop a connector’s consumer group, so they can reset offsets. For Kafka Connect 3.5 or later the new STOP connector API and corresponding button in Lenses can have the same effect.

• Compress schemas before sending them to the Schema Registry. This allows to send larger schemas to the Schema Registry as the limit is on the size of the request rather than the schema itself.

• Improvements to the Skip Validation option for inserting JSON messages, to allow for less strict (but still valid) schemas for inserted messages.

5.5.4

New Features

• Add STOP operation (button) for Connectors. The STOP operation requires Kafka Connect 3.5 or greater

• Allow to skip schema validation when inserting to JSON topics

Improvements

• Connector search is now case-insensitive

• Allow to type to search groups when creating service accounts

• Show masked passwords when editing a connector (regression in 5.5.3)

Fixes

• Filtering connectors by type doesn’t work

• When there were at least two Connect clusters with at least one connector with common name in both clusters, filtering connectors returns incorrect or multiple results

• Validating connectors with passwords may not work (regression in 5.5.3)

5.5.3

New Features

Support for case-insensitive LDAP users

Whilst Lenses users are case-sensitive, LDAP most of the time performs case-insensitive searches on user accounts. This can lead to users who try to login to Lenses with different casing in their username (e.g., user and USER) to get duplicate accounts.

We added the option lenses.security.ldap.case.sensitive with a default value of true. It can be switched to false in which case Lenses will treat usernames from LDAP as case-insensitive and always converting to lowercase.

Improvements

• Upgrade the AWS IAM library to better support service account roles inside EKS

• Upgrade libraries with known CVEs —not affecting Lenses in either way

Fixes

• Fix Grafana link not showing up on sidebar

• Fix a case where some sensitive data might leak in the logs

• Fix filtering by connector name causing the connector screen to crash if a connect cluster is offline

5.5.2

Improvements

• The connectors’ screen will not mask passwords if they are referencing a secret from a secret provider.

Fixes

• Fix regression where connectors’ passwords were not masked.

5.5.1

Improvements

• Authentication:

  • Enhanced authentication to reject with a 401 status code when the user lacks any attached groups in the IdP (Identity Provider).

  • Improved authentication flow, allowing an authenticated SSO (Single Sign-On) user to log in even if there isn’t a corresponding group in Lenses.

• Documentation Enhancement:

  • New SQL processor page with direct links to the latest documentation and support resources for user convenience.

Fixes

• Deployment Issue:

  • Addressed a bug introduced in Lenses git ops deployment* version 5.5.0, resolving provisioning issues experienced in certain deployment scenarios.

• SSO Authentication Fix:

  • Corrected SSO authentication behavior. When an SSO user is configured to overwrite the IdP groups, Lenses now correctly refrains from extracting groups from the IdP.

5.5 Release

New Features

Kafka Connectors as Code

Lenses now introduces support for managing Kafka connectors as code. With this feature, you can define your connectors in a YAML file and seamlessly deploy them to Lenses. This capability is accessible via both the Lenses CLI and the Lenses UI. This release marks the commencement of our journey towards a more declarative and automated approach to managing Kafka and Lenses resources.

Consumer Group Management

In this version, Lenses introduces support for deleting consumer group offsets and entire consumer groups, enhancing flexibility and control over consumer group management.

Generic SSO Provider

Lenses provides support for a few SSO providers out of the box like Google, Okta, etc. In this release, Lenses introduces a generic SSO provider, enabling users to integrate with any SSO provider that supports the SAML 2.0 protocol. This feature is configurable via the lenses.conf file under lenses.security.saml.idp.provider.

Enhancements

Kafka Message Replay

The Kafka message replay feature receives an enhancement, now enabling users to replay messages from a specific offset. This functionality is accessible from both the Lenses topic screen and the Lenses SQL studio screen, providing greater precision in message replay operations.

Consumer Group Offsets Data Link

Users can now seamlessly navigate from the consumer group offsets screen to the data of the topic that the consumer group offset points to, enhancing visibility and ease of data exploration.

Audits to log file

Lenses now provides the capability to log audit events to its log file, enabling users to store audit logs locally for compliance and security purposes. This feature is configurable via the lenses.conf file under lenses.audit.to.log.file.

Lenses Internal Topics Replication Factor

To ensure compatibility with cloud providers such as IBM, where a minimum replication factor is mandated, Lenses now allows the configuration of the replication factor for its internal topics. This setting can be configured in the lenses.conf file under lenses.internal.topics.replication.***.

Bug Fixes

External Applications via Lenses SDK

The Lenses SDK, a thin client facilitating the monitoring and tracking of external applications connected to Kafka within Lenses topology, has been enhanced in this release. An issue where the application’s status in Lenses was not updated correctly has been resolved.

S3 Backup-Restore for JSON Payloads

In this release, a bug affecting the S3 backup-restore feature for JSON payloads has been rectified. Previously, the feature encountered issues due to the Connect converter enforcing schema on JSON payloads, leading to incorrect functionality. This bug has been addressed to ensure seamless backup and restoration of JSON data via S3.

Ugrade Notes

Lenses 5.5 is an incremental release which brings in new features and improvements.

Upgrading from 5.0 or later does not require any static configuration change but if you have automated the creation of any AWS connection, then you will have to adjust the provisioning section of your Helm chart, or your CICD, or —if you use the API directly— your API calls.

Breaking Changes and Caution Items

Lenses upgrades (except patch releases) are not backwards compatible. It is best practice to take a backup of the Lenses database before an upgrade.

New provisioning API [caution]

With Lenses 5.3 the provisioning API was introduced. This new API can be used to create or update the connections landscape. The old provisioning methods could only create the connection landscape (first run).

What this means, is that now the Helm chart or a CICD process can be used to manage Lenses’ connections.

For teams that are on the old provisioning method some adjustments are required to their Helm charts or other provisioning code to switch to the new API. The old methods are still available but are considered deprecated and will be removed or break in the future.

AWS and Glue Connection provisioning [breaking]

With Lenses 5.4 IAM support was added for the AWS connection type. An AWS connection is used as an authentication provider for the Glue Schema Registry and Cloudwatch channels.

Due to this change, if you create or manage your AWS and Glue connections via the API or provisioning, you need to update your configuration to the new format.

Action required

• Add the new authMode property to your connections for AWS and Glue Schema Registry.

Details

• Lenses 5.4 adds a new required property for the AWS and Glue Schema Registry connections.

• The property is authMode.

• It controls how Lenses authenticates with AWS:

  • Access keys (existing feature).

  • Credentials provider chain (new feature).

• You set the property either with the:

  • Connections API - create, update.

  • Provision YAML.

You can set authMode in 2 modes:

1. Access keys mode

This is the existing mode where Lenses uses AWS access keys.

• Set the authMode to Access Key.

• Specify the access key ID and secret access key, as you had before.

2. Credentials provider chain mode (new)

This is the new mode where Lenses uses the AWS default credentials provider chain.

• Set the authMode to Credentials Chain.

• No additional properties needed.

Examples - Provision YAML

1. Access mode

2. Credentials provider chain mode

Examples - API JSON

1. Access mode

AWS connection

Glue Schema Registry connection

2. Credentials provider chain mode

AWS connection

Glue Schema Registry connection

Docker image base change

Starting with Lenses 5.2 the base image of Lenses and SQL Processor Dockers switched from Debian to Ubuntu. On some older systems, these docker images will fail to run, due to a combination of a recent glibc in the container, and older docker daemon on the host.

If you fall under this category, during the startup of the Lenses container, you might see errors such as Unable to identify system. Uname is required or [warning][os,thread] Failed to start thread “GC Thread#0”.

For these cases, we now offer Lenses docker images with the suffix -debian in their tags. E.g:

• lensesio/lenses:5.5-debian

• lensesio/lenses:5.5.0-debian

• lensesio/lenses:latest-debian

If your host is running on an older operating system and you encounter these errors, try to use the debian equivalent tag.

Update Process

Using the Lenses Archive

Make sure you have a JRE (or JDK) installed in the server running Lenses. Lenses can run on JRE 8 or greater, and the recommended version is JRE 11.

Using the Lenses Docker

The docker image uses tags to distinguish between versions. The latest tag (lensesio/lenses:latest) brings the latest stable version of Lenses. There are minor tags to help users get the latest patch in a minor version (e.g 5.5, 5.1) and patch tags to help users pin to a specific patch (e.g 5.5.1, 5.1.2). The best practice advice is to use the minor tag (lensesio/lenses:5.5), which ensures that your installation will always get compatible updates until you made a conscious decision to upgrade the minor version.

If you use the internal database instead of PostgreSQL as the backing store of Lenses, make sure you keep the /data/storage volume to not lose your data. Other volumes supported by the docker are /data/kafka-streams-state which holds state for SQL Processors running IN-PROC and may have to be rebuilt (automatically) if lost, /data/log (log files on disk), /data/plugins (custom UDFs).

Pull the 5.5 docker:

Stop your current container and restart with the 5.5 image, mounting any volumes you might need.

Lenses Box

If you are a Box user, pull the latest version, preserve your /data volume and restart Lenses:

Helm

Download the latest charts and update your values.yaml as described below. Remember that if you are using the internal database instead of PostgreSQL as the backing store, then the Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates. To support a potential downgrade, make sure this volume is backed-up before installing a newer version of Lenses.

If you have provisioning enabled (lenses.provision.enabled: true) in your values.yaml, and you are on provision version “1” then you have to act. Version “1” means either that lenses.provision.version is set to "1", or it is not set at all. You have two options:

• Disable it, as Lenses already has all the information stored in the database, and version “1” does not support updating the connections and license.Copy
• Switch to provisioning version “2” which supports updating connections and licenses every time you do a helm upgrade. To do that, you must make some changes to your old provisioning section. Some resources that can come handy for the switch are:

If you don’t have your values.yaml you can download it from the Kubernetes cluster using Helm:

Proceed to upgrade:

Alternatively, reusing the old values and turning provisioning off:

Cloud Installations

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates. If a new image does not exist, you may be able to update Lenses in-place. Our support team will be happy to go through the available options with you.

\

1. Add your bootstrap brokers including ports

2. Optionally, security protocol e.g. SASL_SCRAM, SASL_SSL

3. Optionally, SASL Mechanism, e.g. SCRAM-SHA-256

SSL/TLS Configuration

If your Kafka connection requires TLS, set the following

• Truststore: The SSL/TLS trust store to use as the global JVM trust store. Available formats are .jks, .p12, .pfx.

• Keystore: The SSL/TLS keystore to use for the TLS listener for Lenses. Available format is .jks.

JMX Metrics

Lenses allows you to connect to brokers JMX. Supported formats are:

1. Simple with and without SSL

2. Jolokia (JOLOKIAG and JOLOKIAP)

   1. With and without SSL

   2. With Basic Auth

   3. Custom http requests and suffix

3. AWS Open Monitoring

Kafka versions

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud.

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

JMX connectivity

Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and OpenMetrics (MSK).

For more enable JMX for Lenses itself see here.

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 6GB RAM/4 CPUs and 500MB disk space.

Memory & CPU

This is the default configuration = Request 1 CPU & Memory 3Gi, Limit 2 CPU & Memory 5Gi

Browser

All recent versions of major browsers are fully supported.

APIs and Websockets

Every action in Lenses is backed by an API or websocket, documented at https://api.lenses.io. A Golang client is available and CLI (command line interface).

Lenses state store

Lenses can use an embedded H2 database or a Postgres database. Postgres is not supplied by Lenses.

TLS termination

By default, Lenses does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Lenses or by using a TLS proxy or load balancer. Refer to the TLS documentation for additional information.

To connect Lenses to your real environment you can:

1. Install Lenses (not the Box) and manually configure the connections to Kafka, Zookeepers, Schema Registries and Connect, or

2. Install Lenses and configure the connections in one go using provisioning.

How to connect to Kafka depends on your Kafka provider.

Lenses can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Follow the guide for your distribution to obtain the credentials and bootstrap broker to provide to Lenses.

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Lenses installation.

Enable Open Monitoring

If you want to have Lenses collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol and set the protocol you want to use

2. SASL Mechanism and set the mechanism you want to use.

Connecting with AWS IAM

In the Lenses bootstrap UI, Select:

1. Security Protocol and set it to SASL_SSL

2. Sasl Mechanism and set it to AWS_MSK_IAM

3. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

4. Optionally upload your trust store

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

Create a Data integration API key

1. From Data integration API keys, select Create Key.

2. For this guide select Global access

Creating a Connection

In the Lenses bootstrap UI, Select:

1. Security Protocol SASL SSL

2. SASL Mechanism PLAIN

In the JAAS Configuration update the username and password from the respective fields Key and Secret of the API key created above:

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster. Lenses can be installed and preconfigured via the .

Edit the relevant Security Group

Enable communications between Lenses & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Lenses installation.

Configure IAM Policies

To authenticate Lenses & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Lenses service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Creating the Connection in Lenses

In the Lenses bootstrap UI, Select:

1. For the bootsrap server configuration, use the MSK Serverless endpoint

2. For the Security Protocol, set it to SASL_SSL

3. Customize the Sasl Mechanism and set it to AWS_MSK_IAM

4. Add software.amazon.msk.auth.iam.IAMLoginModule required; to the Sasl Jaas Config section

5. Set sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler in the Advances Kafka Properties section.

1. During the broker metrics export step, keep it disabled, as AWS Serverless does not export the metrics to Lenses. Click Next

2. Copy your license and add it to Lenses, validate your license, and click Next

3. Click on Save & Boot Lenses. Lenses will finish the setup on its own

Additional Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

Limitations

In our Azure Portal, go to Dashboards > Ambari home.

1. Kafka endpoints: Go to Kafka > Configs > Kafka Broker > Kafka Broker hosts

2. Optionally get the Zookeeper endpoints: Go to Zookeeper > Configs > Zookeeper Server > Zookeeper Server hosts.

In the Lenses bootstrap UI:

1. Set the Kafka endpoints as bootstrap servers

2. Set the security protocol, mechanism and Jaas config according to your setup. For information on configuring clients (Lenses) for your HDInsight cluster see for unauthenticated and for authenticated.

TLS without Authentication

Set the following:

1. security.protocol to SSL

2. Set the password for your trust store

3. Upload your trust store

TLS with Authentication

Perform the additional steps to above

1. Set the password for your key store

2. Upload your key store

3. Set your key password

Lenses integrates with Kafka Connect Clusters to manage connectors.

Adding a connection

To add a connection, go to Admin->Connections->New Connection->Kafka Connect.

1. Provide a name for the Connect cluster

2. Add a comma-separated list of the workers in the Connector cluster, including ports

3. Optionally enable Basic Auth and set the username and password

4. Optionally enable SSL and upload the key-store file

5. Optionally upload a trust store

6. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports

2. Enable basic auth if required and set the user name and password

3. Enable SSL if required and upload the keystore

4. Optionally upload a trust store

5. Set any additional properties

6. Optional enable metrics

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

https://token:{$APIKEY}@{$HOST}/{confluent}

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports, adding the confluent path at the end. Use the value from the kafka_http_url field in the IBM Console Service Credentials tab

2. Enable basic auth if required

3. Set the user name "token"

4. Set the password as the value from API key in the IBM Console Service Credentials tab

Lenses provides support for AWS Glue to manage schema and also explore and process data linked to it via the Lenses SQL Engines.

To connect to Glue, first create a AWS Connection. Go to Admin->Connections->AWS and enter your AWS credentials or select the IAM support if Lenses is running on an AWS host, e.g. EC2 instance or it has the AWS default credentials toolchain provider in place.

Next, Select New Connection->Schema Registry->AWS Glue. Select your AWS Connection with access to Glue, and enter the Glue ARN.

Connectivity to Zookeeper is optional for Lenses. Zookeeper is used by Lenses for such purposes:

1. To provide quotas management (until quotas can be managed via the Brokers API)

2. To autodetect the JMX connectivity settings to Kafka brokers (if metrics are not defined directly for Kafka connection).

To add a Zookeeper connection go to Admin->Connections->New Connection->Zookeeper.

1. Add a comma-separated list of Zookeepers, including port

2. Optionally set a session timeout

3. Optionally set a Chroot path

4. Optionally set a connection timeout

5. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

Lenses can send out alerts and audits events, the following integrations are supported:

Alerts

1. DataDog

2. AWS CloudWatch

3. PagerDuty

4. Slack

5. Alert Manager

6. Webook (Email, SMS, HTTP and MS Teams)

Audits

1. Webhook

2. Splunk

Once you have configure alert and audit connections, you can create alert and audit channels to route events to them. See Monitoring & Alerting or Auditing for more information.

Apicuro supports the following versions of Confluent's API:

• Confluent Schema Registry API v6

• Confluent Schema Registry API v7

Set the schema registry URLs to include the compatibility endpoints, for example:

http://localhost:8080/apis/ccompat/v6

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports and compatibility endpoint path

2. Enable basic auth if required and set the user name and password

Lenses can work with the following schema registry implementations which can be added via the Connections page in Lenses.

Go to Admin->Connections->New Connections->Schema Registry and follow the guide for your registry provider.

Authentication

TLS and basic authentication are supported for connections to Schema Registries.

JMX Metrics

Lenses can collect Schema registry metrics via:

1. JMX

2. Jolokia

Supported formats

• AVRO

• PROTOBUF

To connect your Schema Registry with Lenses, select Schema Registry -> Create Connection.

Schema deletion

To enable the deletion of schemas in the UI, set the following in the lenses.conf file.

## Enable schema deletion in the Lenses UI
## default: false
lenses.schema.registry.delete = true

## When a topic is deleted,
## automatically delete also its associated Schema Registry subjects
## default: false
lenses.schema.registry.cascade.delete = true

Lenses can be deployed in the following ways:

The Lenses docker image can be configured via environment variables or via volume mounts for the configuration files (lenses.conf, security.conf).

Running the Docker

Open Lenses in your , log in with admin/admin and configure your and add your .

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses.conf and security.conf can be mounted directly as

• /mnt/settings/lenses.conf

• /mnt/secrets/security.conf

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

• /data/storage

• /data/plugins

• /data/logs

• /data/kafka-streams-state

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Lenses upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Lenses may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Lenses TLS and Global JVM Trust Store

By default, the Lenses serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

1. Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets

2. Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting Lenses.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.

Lenses uses an AWS in two places:

1. AWS IAM connection to for Lenses itself

3. to Cloud Watch.

What’s in the Box?

Lenses Box contains all components of the Apache Kafka ecosystem, CLI tools, and synthetic data streams.

Starting the Box

2. Install and run the Docker

Kafka Docker advertisement

The broker in the Kafka docker has a broker id 101 and advertises the listener configuration endpoint to accept client connections.

If you run Docker on macOS or Windows, you may need to find the address of the VM running Docker and export it as the advertised listener address for the broker (On macOS it usually is 192.168.99.100). At the same time, you should give the lensesio/box image access to the VM’s network:

If you run on Linux, you don’t have to set the ADV_HOST , but you can do something cool with it. If you set it to be your machine’s IP address, you can access Kafka from any clients in your network.

If you decide to run a box in the cloud, you (and all your team) can access Kafka from your development machines. Remember to provide the public IP of your server as the kafka advertised host for your producers and consumers to access it.

Kafka Docker JMX

Kafka JMX metrics are enabled by default. Refer to ports once you expose the relevant port, ie. -p 9581:9581 you can connect to JMX with

Custom hostname

If you are using docker-machine or setting this up in a Cloud or DOCKER_HOST is a custom IP address such as 192.168.99.100, you will need to use the parameters --net=host -e ADV_HOST=192.168.99.100.

Docker data persistence

To persist the Kafka data between multiple executions, provide a name for your Docker instance and do not set the container to be removed automatically (--rm flag). For example:

Once you want to free up resources, just press Control-C. Now you have two options: either remove the Docker:

Or use it at a later time and continue from where you left off:

Port Numbers

Advanced options

FAQ

How can I run offline?

Download your key locally and run the command:

How much memory to allocate?

The container is running multiple services, and it is recommended to allocate 5GB of RAM to the docker (although it can operate with even less than 4GB).

To reduce the memory footprint, it is possible to disable some connectors and shrink the Kafka Connect heap size by applying these options (choose connectors to keep) to the docker run command:

The AWS Marketplace offering requires AWS MSK (Managed Apache Kafka) to be available. Optionally, AWS RDS (or any other PostgreSQL-compatible database) can be configured for Lenses to store its state.

The following AWS resources are created:

• An EC2 instance that runs Lenses;

• A SecurityGroup to allow network access to the Lenses UI;

• A SecurityGroupIngress for Lenses to connect to MSK;

• A CloudWatch LogGroup where Lenses stores its logs;

• An IAM Role to allow the EC2 instance to store logs;

• An IAM InstanceProfile to pass the role to the EC2 instance;

• Optionally if enabled during deployment: an IAM Policy to allow the EC2 instance to emit CloudWatch metrics.

Deployment takes approximately three minutes.

AWS Marketplace Installation

Select CloudFormation Template, Lenses EC2 and your region.

Choose Launch CloudFormation.

Continue with the default options for creating the stack in the AWS wizard.

Fill in the parameters at Specify stack details.

• Deployment Here the EC2 instance size and password for the Lenses admin user are set. A t2.large instance size is recommended;

• Network Configuration This section controls the network settings of the Lenses EC2 instance. The ingress allows access to the Lenses UI only from particular IP addresses;

• MSK Set the Security Group ID to that of your MSK cluster. A rule will be added to it so that Lenses can communicate with your cluster. You can find the ID by navigating in the AWS console to your MSK cluster and then under Properties -> Networking settings;

• Monitoring Optionally produce the Lenses logs to CloudWatch;

• Storage Lenses stores its state in a database locally on the EC2 instance’s disk or in a PostgreSQL database. Local storage is a development/quickstart option and is not suitable for production use. It is advised to use a Postgres database for smoother upgrades.

Review the stack.

Accept the terms and conditions and create the stack.

Once the stack has deployed, go to the Output tab and click on the FQDN link. If there are no outputs listed you might need to press the refresh button.

Login to Lenses with admin and the password value you have submitted for the parameter LensesAdminPassword.

IAM Support

Lenses supports connection to MSK brokers via IAM. If Lenses is deployed on an EC2 instance it will use the default credential chain loader to authenticate and connect to MSK.

Supported Regions

The following Regions are supported:

• us-east-1;

• us-east-2;

• us-west-1;

• us-west-2;

• ca-central-1;

• eu-central-1;

• eu-west-1;

• eu-west-2;

• eu-west-3;

• ap-southeast-1;

• ap-southeast-2;

• ap-south-1;

• ap-northeast-1;

• ap-northeast-2;

• sa-east-1.

Security Recommendations

Please:

• Do not use your AWS root user for deployment or operations;

• Follow the least privileges principle when granting access to individual IAM user accounts;

• Avoid allowing traffic to the Lenses UI from a broad CIDR block where a more specific block could be used.

Pricing

AWS billing applies for the EC2 instance, CloudWatch logs and optionally CloudWatch metrics.

Troubleshooting

In case you run into problems, e.g. you cannot connect to Lenses, then the logs could provide more information. The easiest route to do this is to go to CloudWatch in the AWS console. Here, find the log group corresponding to your deployment (it has the same name as the deployment) and pick a log stream. The stream with the /lenses.log suffix contains all log lines regardless of the log level; the stream with the /lenses-warn.log suffix only contains warning-level logs.

If the above fails, for example, because the logs integration is broken, you can SSH into the EC2 instance. Lenses is installed into /opt/lenses, the logs can be found under /opt/lenses/logs for further inspection

PLAINTEXT

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

• kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.

• protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL 

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword

SASL_PLAINTEXT vs SASL_SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is if:

1. The transport layer is encyrpted (SSL)

2. The SASL mechanisn for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

Apart from that, when encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

Following are a few examples of SASL_PLAINTEXT and SASL_SSL

SASL_SSL

PLAIN

Encrypted communication and basic username and password for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";      

AWS_MSK_IAM

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

GSSAPI

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword  
    saslMechanism: 
      value: GSSAPI
    saslJaasConfig:
      value: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";      
     keytab:
       file: /path/to/keytab.jks

SASL_PLAINTEXT

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

SCRAM-SHA-256

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";      

SCRAM-SHA-512

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";    

Advanced Client Configuration

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configurationObject:
    kafkaBootstrapServers:
      value:
       - PLAINTEXT://your.kafka.broker.0:9092
    protocol: 
      value: PLAINTEXT
    additionalProperties:
      value:
        isolation.level: "read_committed"
        acks: "all"
        ssl.endpoint.identification.algorithm: "https"

Confluent

Simple configuration, with JMX metrics

The URLs (nodes) should always have a scheme defined (http:// or https://).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      ## all metrics properties are optional
      metricsPort: 
        value: 9581
      metricsType: 
        value: JMX
      metricsSsl: 
        value: false

Basic authentication

For Basic Authentication, define username and password properties.

confluentSchemaRegistry:
- name: schema-registry
  tags: ["tag1"]
  version: 1    
  configuration:
    schemaRegistryUrls:
      value:
        - http://my-sr.host1:8081
        - http://my-sr.host2:8081
    username: 
      value: my-username
    password: 
      value: my-password

TLS with custom truststore

A custom truststore is needed when the Schema Registry is served over TLS (encryption-in-transit) and the Registry’s certificate is not signed by a trusted CA.

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslKeystore:
        fileRef:
          filePath: /path/to/my/keystore.jks
      sslKeystorePassword: 
        value: myPassword

Hard or soft delete

By default, Lenses will use hard delete for Schema Registry. To use soft delete, add the following property:

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      hardDelete:
        value: true      

AWS Glue

Some connections depend on others. One example is the AWS Glue Schema Registry connection, which depends on an AWS connection. These are examples of provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it.

Using AWS Access Key

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Access Key
      accessKeyId: 
        value: my-access-key-id
      secretAccessKey: 
        value: my-secret-access-key
      region: 
        value: eu-west-1
      
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: my-aws-connection
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
          value: arn:aws:glue:region:123123123:registry/my-registry

Using AWS Credentials Chain

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Credentials Chain
      region: 
        value: eu-west-1
        
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:123123123:registry/my-registry

The CLI allows you to import and export resources to and from files.

Import is done on a per-resource basis, the directory structure defined by the CLI. A base directory can be provided by the —dir flag.

Processors, connectors, topics, and schemas have an additional prefix flag to restrict resources to export.

#export
lenses-cli export acls --dir my-dir
lenses-cli export alert-channels --dir my-dir
lenses-cli export alert-settings --dir my-dir
lenses-cli export connections --dir my-dir
lenses-cli export connectors --dir my-dir
lenses-cli export processors --dir my-dir
lenses-cli export quota --dir my-dir
lenses-cli export schemas --dir my-dir
lenses-cli export topics --dir my-dir
lenses-cli export policies --dir my-dir
lenses-cli export groups --dir my-dir
lenses-cli export serviceaccounts --dir my-dir

#import
lenses-cli import acls --dir my-dir
lenses-cli import alert-channels --dir my-dir
lenses-cli import alert-settings --dir my-dir
lenses-cli import connections --dir my-dir
lenses-cli import connectors --dir my-dir
lenses-cli import processors --dir my-dir
lenses-cli import quota --dir my-dir
lenses-cli import schemas --dir my-dir
lenses-cli import topics --dir my-dir
lenses-cli import policies --dir my-dir
lenses-cli import groups --dir my-dir
lenses-cli import serviceaccounts --dir my-dir

Directory structure

The expected directory structure is:

my-dir
├── alert-settings
│   └── alert-setting.yaml
├── apps
│   ├── connectors
│   │   ├── connector-1.yaml
│   │   └── connector-2.yaml
│   └── sql
├── groups
│   └── groups.yaml
├── kafka
│   ├── quotas
│   │   └── quotas.yaml
│   └── topics
│       ├── topic-1.yaml
│       └── topic-2.yaml
├── policies
│   └── policies-city.yaml
├── service-accounts
│   └── service-accounts.yaml
└── schemas
    ├── schema-1.yaml
    └── schema-2.yaml

SQL Processors

Only the update of name, cluster name, namespace, and runner are allowed. Changes to the SQL are effectively the creation of a new Processor.

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582

Custom url mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

• my-kafka-host-0:9582 - because of metricsCustomUrlMappings

• my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings