Lenses is the leading developer experience and UI for exploring and moving real-time data, across any Kafka, on any cloud or on premise. We are on a mission to create an operating fabric to increase developer productivity on real-time data.

Architecture

Lenses is an application that connects to your Kafka environments, allowing you to manage, discover, explore and catalogue your data via SQL. You can also deploy and monitor stream processing applications (SQL Processors) and Kafka Connectors, all wrapped in an enterprize grade RBAC layer.

All Lenses needs is connectivity to your services, think of it as a Kafka client.

The diagram gives a high-level overview of the logical components. At the core of Lenses, we have:

• A Kafka UI for day-to-day work with Kafka

• SQL Engine to query data and create streaming apps leveraging Kafka Streams

• App Engine to manage seamless deployments of SQL apps (deployed to Kubernetes)

• Metadata Engine to create a real-time Data Catalog for cross-system datasets and apps

Lenses is a JVM application exposes secure restful APIs and websockets in addition to providing a Kafka UI. A CLI is available to help automate operations.

Locally on your laptop

For a local quick start, you can use Lenses Box, an all-in-one docker, with Lenses, Kafka, Schema Registry and more. Lenses will start and be configured to connect to the built-in Kafka brokers.

To connect Lenses to your real environment you can:

1. Install Lenses (not the Box) and manually configure the connections to Kafka, Zookeepers, Schema Registries and Connect, or

2. Install Lenses and configure the connections in one go using .

How to connect to Kafka depends on your Kafka provider.

Kafka versions

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud.

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

JMX connectivity

Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and OpenMetrics (MSK).

For more enable JMX for Lenses itself see .

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Increase as a super-user the soft limit to 4096 with:

Use 6GB RAM/4 CPUs and 500MB disk space.

Memory & CPU

This is the default configuration = Request 1 CPU & Memory 3Gi, Limit 2 CPU & Memory 5Gi

Browser

All recent versions of major browsers are fully supported.

APIs and Websockets

Every action in Lenses is backed by an API or websocket, documented at . A Golang client is available and CLI (command line interface).

Lenses state store

Lenses can use an embedded H2 database or a . Postgres is not supplied by Lenses.

TLS termination

By default, Lenses does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Lenses or by using a TLS proxy or load balancer. Refer to the TLS for additional information.

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports

2. Enable basic auth if required and set the user name and password

3. Enable SSL if required and upload the keystore

4. Optionally upload a trust store

5. Set any additional properties

6. Optional enable metrics

Topic settings and naming rules allow for the enforcement of best practices when onboarding new teams and topics into your data platform.

Topic configuration rules

Topic configuration rules can be used to enforce partition sizing, replication, and retention configuration during topic creation. Go to Admin->Topic Settings->Edit.

Naming conventions

By setting naming conventions you can control how topics are named. To define a naming convention, go to Admin->Topic Settings->Edit. Naming rules allow you to select from predefined regex or apply your own.

Searching

In the Explore screen, you can use the search bar to filter for both fields and topic names in your Kafka cluster, additionally, it will search across any Elasticsearch or Postgres instances that have been connected.

Lenses provides support for AWS Glue to manage schema and also explore and process data linked to it via the Lenses SQL Engines.

To connect to Glue, first create a AWS Connection. Go to Admin->Connections->AWS and enter your AWS credentials or select the IAM support if Lenses is running on an AWS host, e.g. EC2 instance or it has the AWS default credentials toolchain provider in place.

Next, Select New Connection->Schema Registry->AWS Glue. Select your AWS Connection with access to Glue, and enter the Glue ARN.

To add descriptions or tags to datasets, click the edit icon in the Summary panel.

Data can be downloaded, optionally including headers, as JSON or as CSV with a choice of delimiters.

To view a configuration for a topic select the Configuration tab. Here you will see the current configurations inherited (default) from the brokers and if they have been overridden (current value).

To edit a configuration click the Edit icon and enter your value.

Apicuro supports the following versions of Confluent's API:

• Confluent Schema Registry API v6

• Confluent Schema Registry API v7

Set the schema registry URLs to include the compatibility endpoints, for example:

http://localhost:8080/apis/ccompat/v6

To add a connection, go to:

Admin->Connections

Select the New Connection button and select Schema Registry.

Enter:

1. Comma-separated list of schema registry URLs including ports and compatibility endpoint path

2. Enable basic auth if required and set the user name and password

1. Add your bootstrap brokers including ports

2. Optionally, security protocol e.g. SASL_SCRAM, SASL_SSL

3. Optionally, SASL Mechanism, e.g. SCRAM-SHA-256

SSL/TLS Configuration

If your Kafka connection requires TLS, set the following

• Truststore: The SSL/TLS trust store to use as the global JVM trust store. Available formats are .jks, .p12, .pfx.

• Keystore: The SSL/TLS keystore to use for the TLS listener for Lenses. Available format is .jks.

JMX Metrics

Lenses allows you to connect to brokers JMX. Supported formats are:

1. Simple with and without SSL

2. Jolokia (JOLOKIAG and JOLOKIAP)

   1. With and without SSL

   2. With Basic Auth

From the IBM Cloud console, locate your bootstrap_endpoints. for the service credentials you want to connect with.

In the Lenses bootstrap UI:

1. Set the bootstrap_endpoints as bootstrap servers

2. Set SASL SSL as the security protocol

3. Set PLAIN as the security mechanism

4. Set the**jaas.conf** as the following, using the apiKey value as the password.

Lenses requires a valid license to start. The license can be added via the UI when in bootstrap mode or at deployment time via the provisioning APIs.

License Expired

If at any point the license becomes invalid (it expired / too many brokers were added to the cluster) only the license page will be available.

License Management

See .

Connectivity to Zookeeper is optional for Lenses. Zookeeper is used by Lenses for such purposes:

1. To provide quotas management (until quotas can be managed via the Brokers API)

2. To autodetect the JMX connectivity settings to Kafka brokers (if metrics are not defined directly for Kafka connection).

To add a Zookeeper connection go to Admin->Connections->New Connection->Zookeeper.

1. Add a comma-separated list of Zookeepers, including port

2. Optionally set a session timeout

3. Optionally set a Chroot path

4. Optionally set a connection timeout

Lenses can send out alerts and audits events, the following integrations are supported:

Alerts

1. DataDog

2. AWS CloudWatch

3. PagerDuty

4. Slack

5. Alert Manager

6. Webook (Email, SMS, HTTP and MS Teams)

Audits

1. Webhook

2. Splunk

Once you have configure alert and audit connections, you can create alert and audit channels to route events to them. See or for more information.

Lenses uses an AWS in two places:

1. AWS IAM connection to MSK for Lenses itself

3. to Cloud Watch.

From the Aiven, locate your Service URI.

In the Lenses bootstrap UI:

1. Set the Service URI as bootstrap servers

2. Set SASL SSL as the security protocol

3. Set SCRAM-SHA-SHA-256 as the security mechanism

4. Set the**jaas.conf** as the following:

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK cluster. Lenses can be installed and preconfigured via the.

Lenses can be deployed in the following ways:

In our Azure Portal, go to Dashboards > Ambari home.

1. Kafka endpoints: Go to

Groups are case-sensitive and mapped by name with Okta

Integrate your user-groups with Lenses using the Okta group names. Create a group in Lenses using the same case-sensitive group name as in Okta.

For example, if the Engineers group is available in Okta, create a group with the same name.

Set up Okta IdP

Lenses is available directly in Okta’s .

Lenses can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Follow the guide for your distribution to obtain the credentials and bootstrap broker to provide to Lenses.

This section guides you through understanding what is required to utilize Lenses efficiently and securely.

Two files control Lenses configuration:

• lenses.conf - contains most of the configuration

• security.conf - sensitive configuration options such as passwords for authentication

All logs are emitted unbuffered as a stream of events to both stdout and to rotating files inside the directory logs/.

The logback.xml file is used to configure logging.

If customization is required, it is recommended to adapt the default configuration rather than write your own from scratch.

The file can be placed in any of the following directories:

• the directory where Lenses is started from

With Basic Auth, user accounts are managed by Lenses and a unique username and a password are used to log in.

Account locking

For BASIC and LDAP authentication type, there is the option to set a policy to temporarily lock the account when successive login attempts fail. Once the lock time window has passed the user can log in again.

.

A Group is a collection of permissions that defines the level of access for users belonging to it. Groups consist of:

• Namespaces

• Application permissions

• Administration permissions

Groups must be pre-created, and the group's names in Lenses must match (case sensitive) those in the SSO provider.

When you first log in to Lenses, use the default credentials admin/admin

The default account is a super user and can be used to create groups and other accounts with appropriate permissions.

The default account username and password may be adjusted as below.

Groups are case-sensitive and mapped to roles, by name, with OneLogin

Integrate your user roles with Lenses using the Keycloak role names. Create a group in Lenses using the same case-sensitive role name as in OneLogin.

For example, if the Engineers role is available in OneLogin, create a group with the same name.

Set up OneLogin IdP

Lenses is available in the OneLogin Application catalog.

Authentication instances too old or in the future

The error that you see

With custom authentication, you can plug in your own authentication system by using HTTP headers.

In this approach, your own authentication proxy/code sitting in front of Lenses takes care of the authentication and injects appropriate Headers in all HTTP requests for verified users.

Setup a custom authentication layer

Set up a custom authentication layer by introducing in security.conf:

Lenses connects similarly to any other application to the infrastructure You can implement a plugin in a few hours in Java/Scala or other JVM technology by implementing one interface:

Kerberos uses SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) for authentication.

Kerberos will automatically log in authorized users when using the /api/auth REST endpoint. If using Microsoft Windows, logging into your Windows domain is usually sufficient to issue your Kerberos credentials.

On Linux, if you use Kerberos with PAM, your Kerberos credentials should be already available to Kerberos-enabled browsers. Otherwise, you will need to authenticate to the KDC manually using kinit at the command line and start your browser from the same terminal.

To view a live snapshot of the metrics for a topic, select the metrics tab for the topic.

This will show you metric information over the last 30 days, on the topic and low JMX metrics.

To view topic partitions select the Partition tab. Here you can see a heat map of messages in the topic and their distribution across the partitions.

Further information about the partitions and replicas is displayed, for example, whether the replicas are in-sync or not.

Inserting a message

To insert a message, select Insert Message from the Action menu. Either enter a message, according to the topic schema or have a message auto-generated for you.

Deleting messages

Deleting messages deletes messages based on an offset range. Select Delete Messages from the Action menu.

Creating schemas

To create a new schema, select New Schema and add your schema.

Viewing schemas

To view the schema associated with a topic, select the Schema tab. Here you can view the schema for both the key and the value of the topic.

Editing Schemas

To edit a schema select either the key or value schema. The schema editor will be expanded, click Edit to change the schema.

Searching for schemas or fields

To list schemas go to Workspace->Schema Registry. Lenses will show the current schemas, you can search in schemas for fields and schema names as well as filtering by format and tags.

Evolving schemas

To evolve a schema, select the schema and select Edit. In the editor apply your changes. If the changes match the evolution rules the changes will be saved and a new version created.

Changing compatibility

To change the compatibility of a schema, select the schema and from the actions menu select Change compatibility.

Enabling approval requests

To enable approval requests, create a group with, or add to a group, the Create Topic Request permission to the data namespace.

Viewing, approving, or rejecting approval requests

Go to Admin->Audits->Requests, select the request, and click view.

Approve or reject the request. If you Approve the topic will be created.

Examining a message

After selecting a topic you will be shown more details of the topic. The SQL Snapshot engine will return to you 200 of the latest messages for partition 0. Both the key and value of the message are displayed in a tree format which is expandable.

At the top of each message, the Kafka metadata (partition, timestamp, offset) is displayed.

Hovering to the right allows you a message the clipboard.

To download all messages to JSON or CSV see here.

Flattening the data view

The SQL Snapshot engine deserializes the data on the backend of Lenses and sends it over the WebSocket to the client. By default, the data is presented in a tree format but it's also possible to flatten the data into a grid view. Select the grid icon.

Searching by partition

Use the partition drop-down to change the partition to return messages you are interested in.

Searching by timestamp

Use the timestamp picker to search for messages from a timestamp.

Searching by offset

Use the offset select to search for messages from an offset.

Live sample

The SQL Snapshot engine has a live mode. In this mode the engine will return a sample of messages matching the query. To enable this, select the Live Sample button. The data view will now update with live records as the are written to the topic. You can also edit the query if required.

Changing the SQL deserializer format

For the SQL Snapshot engine to return data it needs to understand the format of the data in a topic. If a topic is backed by a Schema registry it is automatically set to AVRO. For other types, such as JSON or Strings the engine tries to determine the format.

If you wish to override or correct the format used select either Reset Types or Change Types from the action menu.

Careful monitoring of the data managed by the configured Schema Registry is paramount in order for Lenses to provide its users with the most up-to-date data.

For most cases, this monitoring doesn't cause any issues, but it might happen that, in some cases, Lenses is forced to access the Schema Registry too often.

If this happens, or if you want to make sure Lenses does not go over a rate limit imposed by the Schema Registry, it is possible to throttle Lenses usage of the Schema Registry's API.

In order to do so, it is possible to set the following Lenses configuration:

# schema registry
lenses.schema.registry.client.http.rate.type="sliding" 
lenses.schema.registry.client.http.rate.maxRequests= 200
lenses.schema.registry.client.http.rate.window="2 seconds"

# connect clusters
lenses.connect.client.http.rate.type="sliding"                 
lenses.connect.client.http.rate.maxRequests=200        
lenses.connect.client.http.rate.window="2 seconds"  

Doing so will make sure Lenses does not issue more than maxRequests over any window period.

The exact values provided will depend on things like the resources of the machine hosting the Schema Registry, the number of schemas, how often are new schemas added, so some trial and error is required. These values should however define a rate smaller than the one allowed by the Schema Registry.

Lenses integrates with Kafka Connect Clusters to manage connectors.

Adding a connection

To add a connection, go to Admin->Connections->New Connection->Kafka Connect.

1. Provide a name for the Connect cluster

2. Add a comma-separated list of the workers in the Connector cluster, including ports

3. Optionally enable Basic Auth and set the username and password

4. Optionally enable SSL and upload the key-store file

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

Connections are defined in the provisioning.yaml file. Lenses will then watch the file and resolve the desired state, applying connections defined in the file.

Enabling File Watcher Provisioning

File watcher provisioning must be explicitly enabled. Set the following in the lenses.conf file:

Directory layout

Lenses expects a set of files in the directory, defined by lenses.provisioning.path. The structure of the directory must follow:

1. files/ directory for storing any certificates, JKS files or other files needed by the connection

2. provisioning.yaml - This is the main file, holding the definition of the connections

3. license.json - Your lenses license file

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm, the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime.

Referencing files

Many connections need files, for example, to secure Kafka with SSL you will need a key store and optionally a trust store.

To reference a file in the provisioning.yaml, for example, given:

a file called my-keystore.jks is expected in the files directory. This file will be used for the key store location.

Building on the provisioning.yaml, API provisiong allows for uploading the files directly Lenses from anywhere with network access and without access to the host where Lenses is installed.

Uploading supporting files

Many connections need files, for example, to secure Kafka with SSL you will need a keystore and optionally a trust store.

To reference a file in the, for the configuration option set the key to be "file" and the value to reference in the API request. For example, given:

To upload the file to be used for the configuration option sslKeystore: add the following to the request:

API Call

1. Set the type to application/octet-stream.

2. The name of the part in the multipart request (supporting files) should match the value of the property pointing to the mounted file in the provisioning.yaml descriptor. This ensures accurate mapping and referencing of files.

3. Set LENSES_SESSION_TOKEN as the value of the Lenses Service Account token you want to use to automate provisioning.

In this example, the provisioning.yaml is read from provisioning=@"resources/provisioning.yaml.

The provisioning.yaml contains a reference to "my-keystore-file" which is loaded from @${PATH_TO_KEYSTORE_FILE};type=application/octet-stream

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime. i.e. inject an environment variable from GitHub secrets for passwords.

Authentication is configured in the security configuration file. Lenses Administrator and Basic Auth do not require any configuration.

The following authentication methods are available. Users, regardless of the method need to be mapped to groups.

Account Locking

For BASIC and LDAP authentication types, there is the option to set a policy to temporarily lock the account when successive login attempts fail. Once the lock time window has passed the user can log in again.

These two configuration entries enable the functionality (both of them have to be provided to take effect):

Group Mapping

A Group is a collection of permissions that defines the level of access for users belonging to it. Groups consist of:

• Namespaces

• Application permissions

• Administration permissions

LDAP & Active Directory

When working with LDAP or Active Directory, user and group management is done in LDAP.

Lenses provides fine-grained role-based access (RBAC) for your existing groups of users over data and applications. Create a group in Lenses with the same name (case-sensitive) as in LDAP/AD.

SSO & SAML

When using an SSO solution such as Azure AD, Google, Okta, OneLogin or an open source like KeyCloak user and group management is done in the Identity Provider.

Lenses provides fine-grained role-based access (RBAC) for your existing groups of users over data and applications. Create a group in Lenses with the same name (case-sensitive) as in your SSO group.

Basic Auth

With Basic Authentication, create groups of users and add users to those groups. Authentication and authorization are fully managed, and users can change their passwords.

1. Enable TLS (SSL) for Lenses HTTPS.

2. Create a keystore for SAML.

3. Choose your identity provider (IdP):

Set the following in the security.conf

lenses.security.saml.keystore.location = "/path/to/lenses.p12"
lenses.security.saml.keystore.password = "my_password"
lenses.security.saml.key.password = "my_password"

Groups are case-sensitive and mapped by UUID with Azure

Integrate your user-groups with Lenses using the Azure group IDs. Create a group in Lenses using the UUID as the name.

For example, if the Engineers group has the UUID ae3f363d-f0f1-43e6-8122-afed65147ef8, create a group with the same name.

Set up Microsoft Azure SSO

Learn more about

Add from Azure app-gallery

1. Go to Enterprise applications > + New Application

2. Search for Lenses.io in the gallery directory

3. Choose a name for Lenses e.g. Lenses.io and click Add

4. Select Set up single sign on > SAML

1. Download the Federation Metadata XML file with the Azure IdP details. You will reference this file’s path in the Lenses security.conf configuration file.

Creating topics

To create a topic go to Workspace->Explore->New Topic. Enter the name, partitions and replication factor.

If apply you will not be able to create the topic unless the rules have been met.

Viewing topic details

The Explore screen lists high-level details of the topics

Selecting a topic allows you to drill into more details.

Topics marked for deletion will be highlighted with a D.

Compacted topics will be highlighted with a C.

Increasing the number of partitions

To increase the number of partitions, select the topic, then select Increase Partitions from the actions menu. Increasing the number of partitions does not automatically rebalance the topic.

Overriding a topic configurations

Topics inherit their configurations from the broker defaults. To override a configuration, select the topic, then the Configuration tab. Search for the desired configuration and edit its value.

Deleting topics

To delete a topic, click the trash can icon.

Topics can only be deleted if all clients reading or writing to the topic have been stopped. The topic will be marked for deletion with a D until the clients have stopped.

Empty or compacted topics

To quickly find compacted or empty topics use quick filter checkboxes, for example, you can find all empty topics and perform a bulk delete action on them.

Global Truststore

To use a non-default global truststore, set the path in accordingly with the LENSES_OPTS variable.

LENSES_OPTS=-Djavax.net.ssl.trustStore=/path/to/truststore

Custom Truststore

To use a custom truststore set the following in security.conf. Supported types: jks, pkcs12.

Mutual TLS

To enable mutual TLS, set your keystore accordingly.

Provisioning

A dedicated API, called provisioning, is available to handle bootstrapping key connections at installation time. This allows you to fully install and configure key connections such as Kafka, Schema Registry, Kafka Connect, and Zookeepers in one go. You can use either of the following approaches depending on your needs:

Defining a Connection

Connections are defined in theprovisioning.yaml. This file is divided into components, each component representing a type of connection.

Each component must have:

1. Name - This is the free name of the connection

2. Version set to 1

3. Optional tags

4. Configuration - This is a list of keys/values and is dependent on the component type.

For a full list of configuration options for the connect see .

You can connect Lenses to one or more Kafka Connect clusters. Once connected, Lenses will list the available Connector plugins that are installed in each Cluster. Additionally, Connectors can automatically be and alert notifications sent.

Listing Connectors

To list the currently deployed connectors go to Workspace->Connectors. Lenses will display a list of connectors and their status.

View Connector details

Once a connector has been created, selecting the connector allows us to:

1. View its configuration

2. Update its configurations (Action)

3. View individual task configurations

4. View metrics

View a Connector as Code

To view the YAML specification as Code, select the Code tab in the Connector details page.

Download a Connector as Code

To download the YAML specification, click the Download button.

Creating a Connector

To create a new connector go to Workspace->Connectors->New Connectors.

Select the Connect Cluster you want to use and Lenses will display the plugins installed in the Connect Cluster.

Connectors are searchable by:

1. Type

2. Author

After selecting a connector, enter the configuration of the connector instance. Lenses will show the documentation for the currently selected option.

To deploy and start the connector, click Create.

Create a Connector as Code

Creation of a Connector as code can be done via either

1. Selecting Configure Connector->Configure As Code from the main connector page, or

2. Selecting a Connect Cluster and Connector, then the Code tab

Both options allow for direct input of a Connectors YAML specification or uploading of an existing file.

Managing a Connector's lifecycle

Connectors can be stopped, restarted, and deleted via the Actions button.

Lenses runs as a JVM app; you can tune runtime configurations via environment variables.

Alerts rules are configurable in Lenses, alerts that are generated can then be sent to specific channels. Several different integration points are available for channels.

Infrastructure alerts

These are a set of built-in alerting rules for the core connections, Kafka, Schema Registry, Zookeeper, and Kafka Connect. See infrastructure health.

Data Produced Alerts

Data produced are user-defined alerts on the amount of data on a topic over time. Users have a choice to notify if the topic receives either:

1. more than

2. or less than

Consumer lag alerts

Consumer rules are alerting on consumer group lag. Users can define:

1. a lag

2. on a topic

3. for a consumer group

4. which channels to send an alert to

Application alerts

Lenses allows operators to configure alerting on Connectors. Operators can:

1. Set channels to send alerts to

2. Enable auto restart of connector tasks. Lenses will restart failed tasks with a grace period.

The sequence is:

1. Lenses watches for task failures.

2. If a task fails, Lenses will restart it.

3. If the restart is successful Lenses resets the "restart attempts" back to zero

4. If the restart is not successful, Lenses increments the restart attempts, waits for the grace period and tries another restart if the task is still in a failed state.

Steps 4 is repeated until restart attempts is reached. Lenses will only rest the restart attempts to zero after the tasks have been brought back to a healthy start by manual intervention.

The number of times Lenses attempts to restart is based on the entry in the alert setting.

The restart attempts can be tracked in the page.

Viewing alert events

To view events go to Admin -> Alerts -> Events.

Lenses can work with the following schema registry implementations which can be added via the Connections page in Lenses.

Go to Admin->Connections->New Connections->Schema Registry and follow the guide for your registry provider.

Simple configuration, with JMX metrics

The URLs (workers) should always have a scheme defined (http:// or https://).

This example uses an optional AES-256 key. The key decodes values encoded with AES-256 to enable passing encrypted values to connectors. It is only needed if your cluster uses AES-256 Decryption plugin.

Google doesn't expose the groups, or organization unit, of a user to a SAML app. This means we must set up a custom attribute for the Lenses groups that each user belongs to.

Create a custom attribute for Lenses groups

1. Open the from an administrator account.

Simple configuration, without metrics

Simple configuration, with JMX metrics

The CLI allows you to import and export resources to and from files.

Import is done on a per-resource basis, the directory structure defined by the CLI. A base directory can be provided by the —dir flag.

Processors, connectors, topics, and schemas have an additional prefix flag to restrict resources to export.

Directory structure

The expected directory structure is:

IAM (Identity and Access Management) in Lenses is controlled by Groups. Users and service accounts belong to groups. Permissions are assigned to groups and apply to the users and service accounts in those groups.

Authentication of users is determined by the configured mechanism.

Users must be assigned to a group. SSO and LDAP users are mapped to a group matching the group name provided by the Idp.

Multiple types of users can be supported at the same time.

Service accounts require an authentication token to be authenticated and must belong to at least one group for authorization.

Service accounts are commonly used for automation, for example, when using Lenses CLI or APIs, or any other application or service to interact with Lenses.

Enterprise support is also offered for connectors in the Stream Reactor project, managed and maintained by the Lenses team.

Consumer group monitoring is a key part of operating Kafka. Lenses allows operators to view and manage consumer groups.

It is recommended to install Lenses on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster. Lenses can be installed and preconfigured via the AWS Marketplace.

Edit the relevant Security Group

Enable communications between Lenses & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Lenses installation.

Configure IAM Policies

To authenticate Lenses & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Lenses service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Creating the Connection in Lenses

In the Lenses bootstrap UI, Select:

1. For the bootsrap server configuration, use the MSK Serverless endpoint

2. For the Security Protocol, set it to SASL_SSL

3. Customize the Sasl Mechanism and set it to AWS_MSK_IAM

4. Add

1. During the broker metrics export step, keep it disabled, as AWS Serverless does not export the metrics to Lenses. Click Next

2. Copy your license and add it to Lenses, validate your license, and click Next

3. Click on Save & Boot Lenses. Lenses will finish the setup on its own

Additional Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

More details about how IAM works with MSK Serverless can be found in the documentation:

Limitations

The Lenses docker image can be configured via environment variables or via volume mounts for the configuration files (lenses.conf, security.conf).

Running the Docker

Open Lenses in your , log in with admin/admin and configure your and add your .

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses.conf and security.conf can be mounted directly as

• /mnt/settings/lenses.conf

• /mnt/secrets/security.conf

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

• /data/storage

• /data/plugins

• /data/logs

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Lenses upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Lenses may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Lenses TLS and Global JVM Trust Store

By default, the Lenses serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

1. Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets

2. Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting Lenses.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.

The AWS Marketplace offering requires AWS MSK (Managed Apache Kafka) to be available. Optionally, AWS RDS (or any other PostgreSQL-compatible database) can be configured for Lenses to store its state.

The following AWS resources are created:

• An EC2 instance that runs Lenses;

• A SecurityGroup to allow network access to the Lenses UI;

• A SecurityGroupIngress for Lenses to connect to MSK;

• A CloudWatch LogGroup where Lenses stores its logs;

• An IAM Role to allow the EC2 instance to store logs;

• An IAM InstanceProfile to pass the role to the EC2 instance;

• Optionally if enabled during deployment: an IAM Policy to allow the EC2 instance to emit CloudWatch metrics.

Deployment takes approximately three minutes.

AWS Marketplace Installation

Select CloudFormation Template, Lenses EC2 and your region.

Choose Launch CloudFormation.

Continue with the default options for creating the stack in the AWS wizard.

Fill in the parameters at Specify stack details.

• Deployment Here the EC2 instance size and password for the Lenses admin user are set. A t2.large instance size is recommended;

• Network Configuration This section controls the network settings of the Lenses EC2 instance. The ingress allows access to the Lenses UI only from particular IP addresses;

• MSK Set the Security Group ID to that of your MSK cluster. A rule will be added to it so that Lenses can communicate with your cluster. You can find the ID by navigating in the AWS console to your MSK cluster and then under Properties -> Networking settings;

Review the stack.

Accept the terms and conditions and create the stack.

Once the stack has deployed, go to the Output tab and click on the FQDN link. If there are no outputs listed you might need to press the refresh button.

Login to Lenses with admin and the password value you have submitted for the parameter LensesAdminPassword.

IAM Support

Lenses supports connection to MSK brokers via IAM. If Lenses is deployed on an EC2 instance it will use the default credential chain loader to authenticate and connect to MSK.

Supported Regions

The following Regions are supported:

• us-east-1;

• us-east-2;

• us-west-1;

Security Recommendations

Please:

• Do not use your AWS root user for deployment or operations;

• Follow the least privileges principle when granting access to individual IAM user accounts;

• Avoid allowing traffic to the Lenses UI from a broad CIDR block where a more specific block could be used.

Pricing

AWS billing applies for the EC2 instance, CloudWatch logs and optionally CloudWatch metrics.

For the hourly billed version additional hourly charges apply, which depend on the instance size. For the Bring Your Own License (BYOL) you can get a free trial license .

Troubleshooting

In case you run into problems, e.g. you cannot connect to Lenses, then the logs could provide more information. The easiest route to do this is to go to CloudWatch in the AWS console. Here, find the log group corresponding to your deployment (it has the same name as the deployment) and pick a log stream. The stream with the /lenses.log suffix contains all log lines regardless of the log level; the stream with the /lenses-warn.log suffix only contains warning-level logs.

If the above fails, for example, because the logs integration is broken, you can SSH into the EC2 instance. Lenses is installed into /opt/lenses, the logs can be found under /opt/lenses/logs for further inspection

Integrate your user groups with Lenses using the Keycloak group names. Create a group in Lenses using the same case-sensitive group name as in Keycloak.

For example, if the Engineers group is available in Keycloak, with Lenses assigned to it, create a group with the same name.

Create a new SAML application client in Keycloak

1. Go to Clients

2. Click Create

3. Fill in the details: see the table below.

4. Click Save

Change the settings on client you just created to:

Map user groups

Configure Keycloak to communicate groups to Lenses. Head to the Mappers section.

1. Click Create

2. Fill in the details: see table below.

3. Click Save

Download IdP XML file

Configure in the security.conf file.

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

SSL

Basic Auth

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

AWS

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

Custom url mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

• my-kafka-host-0:9582 - because of metricsCustomUrlMappings

• my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

The following implementations can be specified:

1. Serializers/Deserializers Plug your serializer and deserializer to enable observability over any data format (i.e., protobuf / thrift)

2. Custom authentication Authenticate users on your proxy and inject permissions HTTP headers.

3. LDAP lookup Use multiple LDAP servers or your group mapping logic.

4. SQL UDFs User Defined Functions (UDF) that extend SQL and streaming SQL capabilities.

Once built, the jar files and any plugin dependencies should be added to Lenses and, in the case of Serializers and UDFs, to the SQL Processors if required.

Adding plugins

On startup, Lenses loads plugins from the $LENSES_HOME/plugins/ directory and any location set in the environment variable LENSES_PLUGINS_CLASSPATH_OPTS. These locations Lenses is watching, and dropping a new plugin will hot-reload it. For the Lenses docker (and Helm chart) you use /data/plugins.

Any first-level directories under the paths mentioned above, detected on startup will also be monitored for new files. During startup, the list of monitored locations will be shown in the logs to help confirm the setup.

Whilst all jar files may be added to the same directory (e.g /data/plugins), it is suggested to use a directory hierarchy to make management and maintenance easier.

An example hierarchy for a set of plugins:

SQL Processors in Kubernetes

There are two ways to add custom plugins (UDFs and Serializers) to the SQL Processors; (1) via making available a tar.gz archive at an HTTP (s) address, or (2) via creating a custom docker image.

Archive served via HTTP

With this method, a tar archive, compressed with gzip, can be created that contains all plugin jars and their dependencies. Then this archive should be uploaded to a web server that the SQL Processors containers can access, and its address should be set with the option lenses.kubernetes.processor.extra.jars.url.

Step by step:

1. Create a tar.gz file that includes all required jars at its root:

2. Upload to a web server, ie. https://example.net/myfiles/FILENAME.tar.gz

3. Set

   For the docker image, set the corresponding environment variable

Custom Docker image

The SQL Processors that run inside Kubernetes use the docker image lensesio-extra/sql-processor. It is possible to build a custom image and add all the required jar files under the /plugins directory, then set lenses.kubernetes.processor.image.name and lenses.kubernetes.processor.image.tag options to point to the custom image.

Step by step:

1. Create a Docker image using lensesio-extra/sql-processor:VERSION as a base and add all required jar files under /plugins:

2. Upload the docker image to a registry:

3. Set

   For the docker image, set the corresponding environment variables

Lenses uses Websockets. It can be that your load balancers block them by default. Depending on your load balancer you need to allow websockets.

For example on NGINX:

Lenses can be placed behind a proxy, but you must allow websocket connections.

These two paths are used for WebSocket connections:

• /api/ws

• /api/kafka/ws

Disable proxy buffering for SSE (Server Sent Events) connections on this path:

• /api/sse

TLS termination

Lenses supports TLS termination out of the box, see Enabling TLS

Sample Apache configuration

Sample Caddy configuration

Sample NGINX configuration

To initiate either a topic backup to S3 or topic restoration from S3, follow these steps:

• Navigate to the Actions menu within the Kafka topic details screen.

• Choose your desired action: “Backup Topic to S3” or “Restore Topic from S3.”

• A modal window will open, providing step-by-step guidance to configure your backup or restoration entity.

Identifying if a topic is being backed up

If a topic is being backed up it will be displayed on the topology.

Additional information on the location of the backup can be found by navigating to the topic in the Explore screen where the information is available in the Summary section.

Backing up a topic

To back up a topic, navigate to the topic you wish to back up and select Backup Topic to S3 from the Actions menu.

Enter the S3 bucket ARN and select the that has the Lenses S3 connector installed.

Click Backup Topic, an S3 sink connector instance will now be deployed and configured automatically to back up data from the topic to the specified bucket.

Restoring a topic

To restore a topic, navigate to the topic you wish to restore and select Restore Topic from S3 from the Actions menu.

Enter the S3 bucket ARN and select the Connect Cluster that has the Lenses S3 connector installed. Click Restore Topic, an S3 source connector instance will now be deployed and configured automatically to restore data to the topic from the specified bucket.

Lenses provides monitoring of the health of your infrastructure via JMX.

Additionally, Lenses has a number of built-in alerts for these services.

Monitoring alerts

Lenses monitors (by default every 10 seconds) your entire streaming data platform infrastructure and has the following alert rules built-in:

Broker decommissioning

If you change your Kafka cluster size or replace an existing Kafka broker with another, Lenses will raise an active alert as it will detect that a broker of your Kafka cluster is no longer available. If the Kafka broker has been intentionally removed, then decommission it:

1. Navigate to Services.

2. Select the broker, click on the actions in the options menu and click on the Decommission option.

The JMX endpoint is managed by the lenses.jmx.port option. To disable the JMX leave the option empty.

To enable monitoring of Lenses metrics:

To export via Prometheus exporter:

The Lenses Docker image (lensesio/lenses) automatically sets up the Prometheus endpoint. You only have to expose the 9102 port to access it.

Setting up the JMX Agent with Basic Auth.

This will be done in two parts. The first part is about setting up the required files that JMX Agent will require and the second is about the options we need to pass to the agent.

Lenses state can be stored:

• on the local filesystem - (quick start and default option; deprecated, it will be removed in the next major version)

• in a PostgreSQL database - (recommended) and takes preference when configured

• in a Microsoft SQL Server database

Azure AD supports the LDAP protocol. You can use it as an authentication provider with users, passwords, and groups stored in Azure AD. When a user is authenticated successfully, Lenses queries Azure AD to get the user’s groups and authorizes the user with the selected permissions.

Here is a sample Lenses configuration:

Create Azure AD Domain Instance

1. In the Azure portal create a resource. Search for Domain service