1 of 100

6.1 Overview

Welcome to Lenses, Autonomy in data streaming.

How Lenses Works

Lenses has two components:

HQ

HQ is the control plane / central portal where end users interact with different environments (Kafka clusters). It provides a central place to explore data across many environments.

HQ is a single binary, installed on premise or in your cloud. From HQ you create environments which represent individual Kafka clusters and their supporting services. For each environment, you deploy an agent which connects to Kafka and back to HQ.

Environments

Lenses defines each Kafka Cluster and supporting services, such as Schema Registries and Kafka Connect Clusters, as an environment.

You can have many environments, on premise, in the cloud, provided HQ has network access to the agent and the agent can connect to your Kafka cluster or any Kafka API compatible service.

Each environment has an agent. Environments can also be assigned extra metadata such as tiers, domains and descriptions.

Agents

There's a 1 to 1 relationship between environments, agents and Kafka clusters.

To explore and operate in an environment you need an agent. Agents are headless applications, deployed with connectivity to your Kafka cluster and its supporting services.

Agents only ever communicate with HQ, using an Agent key over a secure channel. You can not, as a user, interact directly with them. End users are unaware of agents, only environments.

Agents require:

Agent Key to establish a communication channel to HQ
Connectivity to a Kafka cluster and credentials to do so.

The agents acts as a proxy to read, or write to your Kafka cluster, execute queries, monitor for alerts and manage SQL Processors and Kafka Connectors.

What's New?

This page details the release notes of Lenses.

Kafka to Kafka Replication

Lenses 6.1 has introduced Kafka-to-Kafka replication, initially supporting AWS MSK to AWS MSK, including Express Brokers. Kafka Replicators can be deployed through the Lenses UI, featuring comprehensive lifecycle management and monitoring capabilities.

Kafka Connections

Kafka Connections let administrators establish links to Kafka using Kubernetes secrets or service accounts that handle connection credentials. Many organizations employ secret providers like AWS Secret Manager or Vault, automatically syncing them to Kubernetes secrets. This process ensures that Lenses or users deploying applications don't need to manage credentials manually.

Environment Creation

A new environment creation flow has been added, so now Lenses Agents can be configured directly from HQ, with a new, in product editor to allow you test can configure Kafka, Schema Registry, Kafka Connect and more connections.

The new APIs support a GitOps style approach, allows you manage the connection state and files fully via the APIs or maintain them in version control.

SQL Studio

The significantly enhanced tree view explorer sidebar in SQL Studio improves user experience when working with an extensive number of topics and environments. We've introduced easier search and navigation designed to minimize scrolling and maximize your daily productivity. Plus, you can now bookmark both topics and environments for quick access.

It is now possible to view a topic's schema directly in SQL Studio, including a powerful split view that allows for seamless comparison of schema versions.

Lenses HQ

Version 6.1.1

Changelog for Lenses 6.1.1

2025-11-27 (YYYY-MM-DD)

Packages

HQ image:
HQ CLI image
Helm charts for HQ and Agent:
Archive installation:

Features / Improvements & Fixes

Changelog details for Lenses 6.1.1

New 🎉

SQL Studio

Left panel with explorer toggle had been removed.
Schema Registry entries can now be listed and the schema details viewed from the Studio.
Context menus are now available in both explorer and breadcrumbs.

Improvements 💪

K2K

When a new K2K app is created, it automatically sets the Docker image to K2K Kafka-to-Kafka replicator version 1.1.0.

K2K IAM

The IAM checks for the K2K application have been enhanced to include dependencies such as source and target environment Kafka connections, Kubernetes cluster, and namespace. This update ensures comprehensive permission validation across multiple axes.

New actions introduced:

ManageOffsets: Allows management of K2K application resources.
GetKafkaConnectionDetails: Retrieves details for Kafka connection resources.

UI

Various fixes have been applied throughout the user interface to address glitches and inconsistencies, enhancing the in-app user experience.

Version 6.1.0

Changelog for Lenses 6.1.0

2025-10-24 (YYYY-MM-DD)

Packages

HQ image:
- lensesio/lenses-hq:6.1.0
HQ CLI image
- lensesio/lenses-cli:6.1.0
Helm charts for HQ and Agent: https://lenses.jfrog.io/ui/native/helm-charts
Archive installation: https://archive.lenses.io/lenses/

Features / Improvements & Fixes

Changelog details for Lenses 6.1.0

New 🎉

Kafka Connections

Kafka Connections allow administrators to define connections to Kafka as Kubernetes secrets or service accounts, that reference the credentials to connect with.

Most organisations already use secret providers such as AWS Secret Manager or Vault, and sync these to Kubernetes secrets automatically. This ensures Lenses or users deploying applications never need to deal with the credentials themselves.

See the documentation to get started.

Kafka to Kafka Replication

Lenses Kafka to Kafka Replicator is now integrated into Lenses. You can configure and deploy Lenses, with predefined Kafka Connections to move data between AWS MSK IAM clusters.

More cluster authentication methods and providers coming soon!

See the documentation to get started.

Improvements 💪

Configure Agent Provisioning from HQ

You can now create an environment and configure the Agent provisioning directly from HQ. JSON schema support is added, providing syntax highlighting, auto completion and error reporting.

You can also view and edit the existing provisioning files of agents already connected.

SQL Studio

SQL Studio is moving forward again, and now brings a more IDE style experience. An improved tree navigation provides:

Improved search functionality
Expanded topics nodes to browse schemas and consumer groups associated with topics
Context menu support to allow actions on topics
Bookmarking of favourite topics.

Performance

Enhanced the performance of the environments screen, making it more responsive and capable of handling larger datasets.

Helm

Service Type Configuration: Added configurable service.type parameter to allow customization of the Kubernetes service type (ClusterIP, NodePort, LoadBalancer, etc.)

New parameter in values.yaml: service.type (default: ClusterIP)
Updated service template to use the configurable value

Fixes ✅

Helm

Fix values.schema.json examples format for postgres params object

Lenses Agent

6.1.1

Changelog for Lenses 6.1.1

2025-11-27(YYYY-MM-DD)

Packages

Agent image:

```
lensesio/lenses-agent:6.1.1
```
Helm charts for HQ and Agent: https://helm.repo.lenses.io/
Archive installation: https://archive.lenses.io/lenses/

Features / Improvements & Fixes

Changelog details for Lenses 6.1.1

Improvements 💪

SQL Processor Docker image

When developing a SQL processor, the application Docker image is fixed. Consequently, any updates for performance or security enhancements cannot be applied. In this release, a new API and user interface options were introduced to enable attachment of the desired Docker image.

Fixes 🛠️

Kafka Quotas

Fixes an issue preventing default quotas for Users and Clients to be properly stored and applied.

6.1.0

Changelog for Lenses 6.1.0

2025-10-24(YYYY-MM-DD)

Packages

Agent image:

```
lensesio/lenses-agent:6.1
```
Helm charts for HQ and Agent: https://helm.repo.lenses.io/
Archive installation: https://archive.lenses.io/lenses/

Features / Improvements & Fixes

Changelog details for Lenses 6.1.0

Improvements 💪

Minimal start up requirements

The Agent now requires only a connection to HQ for startup. This connection can be configured through environment variables or by setting a provisioning file with HQ details. This streamlined process enhances the user experience for creating a new environment, allowing Lenses HQ to automatically push connection details to the Lenses agent.

Logging

Enhance the logs outlining the HQ connectivity status.

Helm

Service Type Configuration: Added configurable service.type parameter to allow customization of the Kubernetes service type (ClusterIP, NodePort, LoadBalancer, etc.)

New parameter in values.yaml: service.type (default: ClusterIP)
Updated service template to use the configurable value

Add persistence.provisioning configuration with 50Mi default size, disabled by default

Create PVC template for provisioning data storage at /data/provisioning
Update deployment to mount provisioning volume when enabled
Add helper function for provisioning claim name generation
Add tests for provisioning volume
Switch .Values.persistence.existingClaim to .Values.persistence.log.existingClaim and .Values.persistence.provisioning.existingClaim and fail the deployment if the old value is being used

Fixes ✅

Helm

Component Label Correction: Fixed component labels from lenses to lenses-agent for proper identification

Updated in deployment.yaml
Updated in service.yaml

Getting Started

Setting Up Community Edition

This quick start guide will walk you through installing and starting Lenses using the Community Edition, an all-in one docker compose, including Kafka brokers.

This is a quick start for a local setup using the Lenses Community edition. To connect to your Kafka clusters, see here.\

By running the following command, including the ACCEPT_EULA, you are accepting the Lenses EULA agreement.

Run the following command:

The very first time you run this command it will take a bit longer as Docker has to download the images. Subsequent runs should take much less time.

To run this setup smoothly, your Docker settings must allocate at least 5GB of memory

Once the images are pulled and containers started, you can log in by going to http://localhost:9991 or the IP of your Docker host.

Username: admin 
Password: admin

The HQ binary does not have a default password. This is a default password configured by the Docker Compose scripts to secure your deployment.

CHANGE THE DEFAULT PASSWORD. You can see how here.

It may take a few seconds for the agent to fully boot and connect to HQ.

You will need an access code to use Community Edition. It will ask you to set it up the first time you login. Once applied you won't need to access it again. Please see the self-guided walk through for details on setting it up.

The quick start uses a docker-compose file to:

Start Postgres, HQ and Agent uses Postgres as a backing store.

Start a single local Kafka Broker

Start a local Confluent Schema Registry

Start HQ and create an environment & agent key

Start Agent and connect to Kafka, the Schema Registry and HQ.

Hands-On Walk Through of Community Edition

Simple walk through to introduce you to the Lenses 6 user interface.

Outline of Walk Through

Logging in to Community Edition
Exploring Lenses UI
Adding a new environment
Searching Topics and Schemas
Using SQL Studio
Drilling Down Into Environments
Adding a Data Policy

1. Logging in to Community Edition

After you've run your docker compose command you can access it running locally at . CE will ask you to login:

User: admin Pass: admin

The very first time you login Lenses will ask you to verify with your email. This is easy to setup, just click on the "Verify" button:

If you have done the verification before, then you can enter your email address and the received access code from the link "Already verified?":

The verify link will take you to the setup page on Lenses website, where you can enter your email address:

Click Send Link and then Lenses will send you an email with a magic link to activate Lenses Community Edition. Be sure to check your junk folder if it doesn't arrive. In this email you will also find other important information - your personal access code and useful links, which will help you to quick start with Lenses. Don't forget to bookmark and keep this email.

The very first time you login to Lenses CE you will see our first start help screen. There is a video to watch as well as links to these docs, , our and .

2. Exploring Lenses UI

Click on Let's Start to access the Lenses UI. The first view you'll see is the Environments view. This is where Lenses displays all of your connected Kafka Environments. This can include: the Kafka clusters themselves, Kafka Connect, Schema Registry, connectors, consumers, even the Kubernetes clusters it's all running in. Environments mean your entire Kafka ecosystem not just the clusters themselves. For our demo setup we only have one environment connected, but you can have up to two at no charge with Community Edition.

Click on the link below Environments view to switch to the topics view. Here you'll see all the topics in your connected Environments. We are currently logged in as Admin so we can see all the Topics in our Environments. If we were logging in with a more restricted role we might only see the Topics we have permission to view.

Use the bottom scroll bar to scroll to the right so you can see further information about each topic.

You can see what type of schema it uses, how many partitions it uses, and much more.

3. Adding a new environment

Adding a new environment will allow you to connect a second Kafka cluster to Lenses HQ.

Important! Before you begin, ensure you have your Kafka Cluster already deployed and all its credentials. You will need them to configure the Lenses Agent. Without them, it will not be able to connect to your environment.

Click on the button New environment in the top right corner and the new environment wizard will guide you through the process.

Fill the environment name and give it a short description, if you like. You can select also domain membership, if you use the Domain conventions and the environment tier. After you are ready, click on the button "Create environment"

This will generate an Agent key. You need this to start the agent docker.

Copy the command to start the docker, this uses the environment variables to create a provisioning.yaml to connect to HQ.

It may take a minute for Agent to start fully and connect.

By default the agent will create a mounted volume. We recommended once the agent is full configured you download the provisioning.yaml for safe keeping.

Once the agent docker has started, its will connect and you can move onto the next step, select the type of Kafka and other services you need. This will update the YAML editor and highlight any errors. You can also type directly in the editor to get JSON schema support is required, e.g. type "Kafka" to get the default snippets.

Once you have entered the details for your Kafka and have no validation errors you can test the configuration. This will push the configuration to the agent, where it will check validation and connectivity.

If valid, you can apply to the agent.

By testing the configuration first you will get feedback in what will change, be deleted or added.

If the configuration fails, errors and the line number will be visible in the problem panel.

4. Searching Topics and Schemas

The topics view is fully searchable. So for example if we wanted to build a "Customer Location View" for our web page using Kafka data — we could search for the keyword longitude here and see which topics include location data. Let's do a search for "latitude" in the topics view and see what comes up:

Three topics appear to have data about latitude, but let's dive a bit deeper. Tick on the "Search in Schema" tickbox to get Lenses to display the actual names of the keys in the schema.

This will surface the actual schema names that match your search.

5. Using SQL Studio

Based on what we've discovered it seems like the nyc_yellow_taxi_trip_data might be useful for our theoretical project. Let's use Lenses to dive a bit deeper into that topic and view the actual data flowing through using SQL Studio. To get to SQL Studio from this view simply hover your mouse over the topic: nyc_yellow_taxi_trip_data. That will cause the interactive controls to appear. Click on the SQL shortcut when it pops up:

Clicking that button automatically opens up that topic in SQL Studio. You can now interact directly with the data flowing through that topic using SQL statements. Note when you first access SQL Studio it appears with both side "drawers" open. You can click on the drawer close icons on either side to make more room to work directly with your data and SQL.

Now you can go back and open those as needed later on, but this give you all the screen to view and work with your data. Toggle your view from Grid to List. Now you have your data in a more JSON style format. Expand out the JSON to view the individual key / value pairs. Across the top you'll see the metadata for each event: Partition, Offset, and Time Stamp. Below you can examine the key / value pairs. As you can see we've got plenty of longitude and latitude data to work with for our customer location visualization.

Now let's move on from data discovery to troubleshooting. Using the same taxi data topic we can troubleshoot a "live" problem. Several drivers are reporting errors with credit card transactions going through in the last 15 minutes. Let's use SQL Studio to examine taxi transactions in the last 15 minutes using a SQL search:

Copy that text and paste it into the SQL box in your SQL Studio. Then from the Time Range picker select the last 15 minutes to set your time frame and then hit run.

Next up let's clean up our view so the data is a bit easier to see. Go to the Columns button and get ride of the timestamp, partition, and offset columns. Now we just have our vendorID, fare_amount, and payment_type. Assuming payment_type = 1 means the customer paid cash and payment_type = 2 means card scroll down and notice that both types of payments seem to be going through. Maybe the problem is with a particular driver. Let's filter our results on vendorID. Select the Filters button and create a filter to just show vendorID = 1.

Toggle the filter back and forth between vendorID = 1 and 2 and see that transactions of both types seem to be flowing through. So perhaps the drivers' reported problem is not here, maybe it's a wireless connectivity issue? We could check our wireless telecom topic to further troubleshoot this theoretical issue. We have a detailed guide to Lenses SQL and using SQL Studio in our docs here:

6. Drilling Down Into Environments

But for now let's move on to more other Lenses features. Let's switch back to the Environments View. Hover your mouse over our environment and some controls should appear. Click on the big arrow that appears in order to drill down into the specifics for this environment.

Now we're in the details page of this specific Environment. We can quickly see the health of all the components of the Kafka ecosystem. We can use any of the specific views on the left side, or drill down more interactively from details that appear on the main dashboard. Take a moment to look around at all the stats and data presented on this page before we move on.

On the lefthand side switch to the Topics view and select the backblaze_smart topic. That will open up the Topic View. Here we can see examples of the data but can also view much more detailed information about the topic. Be sure to click on the button to close the right side drawer to free up some screen space. Take a moment to toggle through the different topics view as listed across the top but then come back to the Data view.

7. Adding a Data Policy

Coming back to the Data View you'll notice that we have serial_number field displayed. This field is tied to registered owners and can be considered personally identifiable data. Luckily Lenses has the capability to block the view of this sensitve data. We need to setup a Data Policy to block this. Make a note of the name of the filed we want to obscure: serial_number.

Click on the Policy view on the left hand side and click on New Policy. Then fill out the form:

Name: serial-number-blocker

Redaction: last 3 (this means we'll mark out everything but the last 3 digits in the number)

Category: private_info (note after you type this in you'll need to hit enter to make it stick)

Impact Type: medium

Affected Datasets: don't change

Add Fields: serial_number (you'll need to hit return here as well to make it stick)

Once you're done it should look like this:

Then click "Create New Policy"

Now you'll see your new policy in the list. You can go back to the topics page and click on backblaze_smart topic again and verify that the serial_number field has been obfuscated.

It should look like this:

Congrats you've completed a basic introduction to Lenses 6. There's lots more to learn and features to use. Look for more tutorials to come soon.

Connecting Lenses to your Kafka environment

Connect Lenses to your actual environments (Kafka clusters).

Overview

This page describe an overview of deploying Lenses against your Kafka clusters.

This guide walks you through manually deploying HQ and an Agent to connect to your Kafka clusters. Lenses acts as a Kafka client, it can connect to any provider exposing a Kafka-compatible API.

For more detailed guides on Helm, Docker, and Linux, see here.

How to connect to your Kafka?

To deploy Lenses against your environments, you need to:

Configure HQ

Optionally use your own Postgres instance.

Start HQ

Start HQ using your configuration from step 1.

Create an Environment in HQ

To connect an agent to HQ for your Kafka cluster, we need to create an environment in HQ.

Start & Configure an Agent

To configure the agent, you need to:

Optionally using your own Postgres instance, it uses an embedded database by default
Start it with the Agent Key from step 3
Configure a connection to your Kafka cluster

Prerequisites

EULA acceptance

To start HQ and an Agent, you have to accept the Lenses EULA.

For HQ, in the config.yaml set:

config.yaml

license:
  acceptEULA: true

Kafka

Any version of Apache Kafka (2.0 or newer) on-premise and in the cloud. Supported providers include:

Confluent Platform & Cloud
AWS MSK & AWS MSK Serverless
Aiven
IBM Event Streams
Azure HDInsight & EventHubs

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

Postgres

Only needed if you want to use your Postgres. The docker compose will start a local Postgres instance.

HQ and Agents can share the same instance, by either using a separate database or schema for HQ and each agent, depending on your networking needs.

Postgres server running version 9.6 or higher.

Database Role

The recommended configuration is to create a dedicated login role and database for the HQ and each Agent, setting the HQ or Agent role as the database or schema owner. Both the agent and HQ need credentials; create a role for each.

terminal

# login as superuser and add Lenses role and database
psql -U postgres -d postgres <<EOF
CREATE ROLE lenses_agent WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_agent OWNER lenses_agent;

CREATE ROLE lenses_hq WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_hq OWNER lenses_hq;
EOF

Networking

Web sockets - You may need to adjust your load balancer to allow them. See here.
JMX connectivity - Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are supported, including JOLOKIA and Open Metrics (MSK).

For more enable JMX for the Agent itself, see here.

Kafka ACLs

These ACLs are for the underlying Lenses Agent Kafka client. Lenses has its own set of permissions guarding access.

You can restrict the access of the Lenses Kafka client but this can reduce the functionality on offer in Lenses, e.g. not allow Lenses to create a topic at all, even though this can be managed by Lenses own IAM system.

The agent requires access to your Kafka cluster. If ACLs are enabled, you will need to allow the Agent access.

SSO (optional)

If you want to use SSO / SAML for authentication, you will need the metadata.xml file from your provider. See Authentication for more information.

Install

This page describes configuring and starting Lenses HQ and Agent against your Kafka cluster.

This guide uses the Lenses docker-compose file. For non-dev installations and automation see the section.

Configure HQ

HQ is configured via by one file, config.yaml. The docker-compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

You only need to follow this step if you do not want to use the local Postgres instance started by the docker-compose file.

You must create a database and role in your Postgres instance for HQ to use. See .

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

Authentication

Currently HQ supports:

Basic Authentication (default)
SAML

For this example, we will use basic authentication, for information on configuring other methods, see and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

You can now log in to your with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

Login into HQ and create an environment, Environments->New Environment.
At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

Configure the Agent

The Agent is configured via two files:

lenses.conf - holds low-level options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file
provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

You only need to follow this step if you do not want to use the local Postgres instance started by the docker-compose file.

You must create a database and role in your Postgres instance for the Agent to use. See .

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

Connect the Agent to HQ

You can connect the agent to HQ in two ways, all via

Start the Agent docker with the an AGENT_KEY via environment variables at minimum. You need to create an environment in HQ to get this key
Or mount a provisioning file that contains the connection to HQ, recommended for TLS enabled HQs

You can still reference environment variables if you mount the file, e.g

Minimal Start

First deploy HQ and create an environment, then with the AGENT KEY run:

This will start and connect to HQ but not to Kafka or other services. It will create a provisioning file in data/provisioning.

Set the docker network accordingly

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this, update the provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

You can add connections in three ways:

direct editing of the provisioning file directly
Lenses UX
APIs (which step 2 uses)

They all result in writing a provisioning file which the Agent picks up and loads.

Manual file editing

You must manual add all the connections you want to the file and then mount it. To help you create a provisioning file you can use the JSON SCHEMA support. In you IDE, like VS Code, create a file called provisioning.yaml and add the following line at the top:

Then start typing, for example k for Kafka and Kafka Connect, s for schema registry or just crtl+space to trigger the default templates.

Fill in the required fields, your editor should highlight issues for you

Start with provisioning

Add lenses-agent.conf if you are overriding defaults like the embedded database.

See for examples of different authentication types for Kafka.

Lenses UX

When you create an environment via Lenses UI, you will be guided through the process to, start the agent, and configuration the connections. The experience is similar to manually editing the provisioning but it uses the APIs to push down and test configurations.

APIs

You can also use the APIs directly. See .

Deployment

Installation

This page describes the supported installation methods for Lenses.

Lenses can be deployed in the following ways:

Kubernetes - Helm

This page describes installing Lenses HQ and Agent in Kubernetes via Helm.

Only Helm 3 is supported.

Docker

This page describes installing Lenses with Docker Image.

Deploying HQ

This page describes deploying Lenses HQ via docker.

The HQ docker image can be configured via volume mounts for the configuration file.

The HQ looks for the config.yaml in the current working directory. This is the root directory for Docker.

Running the Docker

terminal

docker run --name lenses-hq \
--network panoptes \
-p 8080:8080 \
-v $(pwd)/config.yaml:/config.yaml\
lensting/lenses-hq:6-preview

Prerequisites

The main pre-requirements that has to be fulfilled before Lenses HQ container can be started and those are:

Postgres database
- See docs here for information on configuring Postgres to work with Agent.

Complete configuration file

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

Main configuration file that has to be configured before running docker command is config.yaml.

Assertion Consumer Service endpoint is following

/api/v2/auth/saml/callback?client_name=SAML2Client

Sample configuration file is following:

config.yaml

auth:
  administrators:
    - admin
    - [email protected]
  users:
    - username: admin
      # bcrypt("correcthorsebatterystaple").
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG
  sessionDuration: 24h
  saml:
    enabled: true
    baseURL: https://lenses6.company.com
    entityID: https://lenses6.company.com
    metadata: |          
         <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor
          xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"
            </md:IDPSSODescriptor>
          </md:EntityDescriptor>
          
    userCreationMode: manual
    groupMembershipMode: manual
    uiRootURL: /
    groupAttributeKey: groups
    authnRequestSignature:
      enabled: false
http:
  address: :8080
  accessControlAllowOrigin:
    - https://lenses6.company.com
  accessControlAllowCredentials: false
  secureSessionCookies: false
  
agents:
  address: :10000
  
database:
  host: postgres-postgresql.postgres.svc.cluster.local:5432
  username: $(LENSESHQ_PG_USERNAME)
  password: $(LENSESHQ_PG_PASSWORD)
  schema:
  database: lenseshq
  TLS: false
license:
  key: license_key_
  acceptEULA: true
logger:
  mode: text
  level: debug
metrics:
  prometheusAddress: :9090

More about configuration options you can find on the HQ configuration page.

What's next?

After the successful configuration and installation of HQ, the next steps would be:

Deploying and Agent
Configuring IAM roles / groups / policies

Linux

This page describes install the Lenses via a Linux archive

Deploying an Agent

This page describes the install of the Lenses Agent via an archive on Linux.

To install the Agent from the archive you must:

Extract the archive
Configure the Agent
Start the Agent

Extracting the archive

Installation link

Link to archives can be found here:

Extract the archive using the following command

Inside the extract archive, you will find.

Configure the Agent

To configure the agents connection to Postgres and its provisioning file. See here in the .

Once the agent files are configure you can continue to start the agent.

The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.

Connect the Agent to HQ

You can connect the agent to HQ in two ways, all via

Start the Agent with the an AGENT_KEY via environment variables at minimum. You need to create an environment in HQ to get this key
Or provisioning file that contains the connection to HQ, recommended for TLS enabled HQs

You can still reference environment variables if you use the file, e.g

Database

My default the Agent will start with an embedded database, if you wish to use Progress, recommended for production, see . Database settings are set in lenses-agent.conf

Starting the Agent

Provisioning file path

If you configured provisioning.yaml make sure to place following property:

Start Lenses by running:

or pass the location of the config file:

If you do not pass the location of lenses-agent.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

In case agent fails with error message that security.conf does not exist and is provided just run following command under lenses directory

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the lenses-agent.conf to be readable only by the lenses user.

The agent needs write access in 4-5 places in total:

[RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:
1. [RUNTIME DIRECTORY]/logs Where logs are stored
2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.
3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.
4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.
5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

Back-up this location for disaster recovery

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

Mount /tmp without noexec
or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Increase as a super-user the soft limit to 4096 with:

Use 8GB RAM /4 CPUs and 20GB disk space.

Configuration

This page describes how to configure Lenses.

Authentication

This page describes the authentication methods supported in Lenses.

Authentication is configured in HQ.

Users can authentication is two ways. Basic authentication and SSO / SAML. Additional specific users can be assigned as admin accounts.

Admin Account

This page describes how to configure admin accounts in Lenses.

You can configure a list of the principals (users, service accounts) with root admin access. Access control allows any API operation performed by such principals. If not set, it will default to [].

Admin accounts are set in the config.yaml for HQ under the key, as an array of usernames.

Changing the Admin Password

To change the admin password, update the config.yaml in the section. Set the password of the admin users.

Passwords need to be bcrypt hashes.

You can use the Lenses CLI to create a bcrypt password. You can download the CLI . The executable for Lenses 6 CLI is called "hq".

Basic Authentication

This page describes configuring basic authentication in Lenses.

Basic authentication is set in the config.yaml for HQ under the http.users key, as an array of usernames and passwords.

Passwords need to bcrypt hashes.

This ensures that the passwords are hashed and secure rather than stored in plaintext. For instance, instead of using "builder" directly, it should be hashed using bcrypt.

An example of a bcrypt-hashed password looks like this:

$2a$12$XQW..XQrtZXCvbQWertqQeFi/1KoQW4eNephNXTfHqtoW9Q4qih5G.

Always ensure that you replace plaintext passwords with their bcrypt counterparts to securely authenticate users.

You can use the Lenses CLI to create a bcrypt password:

lenses utils hash-password

config.yaml

auth:
  users:
  - username: bob
    password: $2a$12$XQW..XQrtZXCvbQWertqQeFi/1KoQW4eNephNXTfHqtoW9Q4qih5G 
  - username: brain
    password: $2a$12$XQW..XQrtZXCvbQWertqQeFi/1KoQW4eNephNXTfHqtoW9Q4qih5G

SSO & SAML

This page describes configure SSO & SAML in Lenses for authentication.

Overview

This page gives an overview of SSO & SAML for authentication with Lenses.

Users

Control of how user create with SSO is determined by the SSO User Creation Mode. There are two modes:

Manual
SSO

With manual mode, only users that pre-created in HQ can login.

With sso mode, users that do not already exists are created and logged in.

Group Mapping

Control of how a user's group membership should be handled in relation to SSO is determined by the SSO Group Membership Mode. There are two modes:

Manual
SSO

With the manual mode, the information about the group membership returned from an Identity Provider will not be used and a user will only be a member of groups that were explicitly assigned to them in HQ.

With the sso mode, group information from Identity Provider (IdP) will be used. On login, a user's group membership is set to the groups listed in the IdP.

Groups that do not exist in HQ are ignored.

SAML configuration is defined in the config.yaml provided to HQ. For more information on the configuration options see here.

config.yaml

http:
  saml:
    metadata: |-

The follow SSO / SAML providers are supported.

Creating a Keystore

Enable SAML single-sign on by creating a keystore.

SAML needs a keystore with a generated key-pair.
SAML uses the key-pair to encrypt its communication with the IdP.

Creating a keystore

Use the Java keytool to create one.

keytool \
 -genkeypair \
 -storetype pkcs12 \
 -keystore lenses.p12 \
 -storepass my_password \
 -alias lenses \
 -keypass my_password \
 -keyalg RSA \
 -keysize 2048 \
 -validity 10000

Setting

Definition

storetype

The type of keystore (pkcs12 is industry standard, but jks also supported)

keystore

The filename of the keystore

storepass

The password of the keystore

alias

The name of the key-pair

keypass

The password of the key-pair (must be same as storepass for pkcs12 stores

Azure SSO

This page describes configuring Azure SSO for Lenses authentication.

Learn more here about

Add Lenses from Azure SSO gallery

Go to Enterprise applications->New Application.

Search for Lenses.io in the gallery directory.

Choose a name for Lenses e.g. Lenses.io and click Add.

Enable SSO

On the overview page select Single Sign On.

Configure SAML

Remember to activate HTTPS on HQ. See .

Setting

Value

Download SAML Certificates

Download the Federation Metadata XML.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See for more details.

Okta SSO

This page describes configuring Okta SSO for Lenses authentication.

Set up Okta IdP

Lenses is available directly in Okta’s Application catalog.

Add application in the Catalog

Go to Applications->Applications
Click Add Application
Search for Lenses
Select by pressing Add

Set General Settings

App label: Lenses
Set the base url of your lenses installation e.g. https://lenses-dev.example.com
Click Done

Download SAML Certificates

Download the Federation Metadata XML file with the OktaIdP details.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

OneLogin SSO

This page describes configuring OneLogin SSO for Lenses authentication.

Set up OneLogin Id

Lenses is available in the OneLogin Application catalog.

Visit OneLogin’s Administration console.

Select Applications->Applications->Add App
Search and select Lenses
Optionally add a description and click save

Add Lenses via the Application Catalog

In the Configuration section set the base path from the url of the Lenses installation e.g. lenses-dev.example.com (without the https://)
Click Save

Download SAML Certificates

Download the Federation Metadata XML file with the OneLogin IdP details.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See for more details.

Generic SSO

This page describes configuring a Generic SSO provider for Lenses authentication.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See for more details.

Agent

This page describe the Lenses Agent configuration.

Overview

This page describes an overview of the Lenses Agent configuration.

The Agent configuration is split between two files.

lenses-agent.conf
provisioning.yaml

lenses-agent.conf holds all the database connections and low-level options for the agent.

In the you define how to connect to your Kafka cluster, Schema Registries, Kafka Connect clusters and HQ. See for more information.

The provisioning.yaml is watched by the Agent, so any changes made, if valid, are applied.

JSON Schema Support

To help with creating a provisioning.yaml from your IDE you can use the provided JSON schema support. They are available in the following repo.

Add the following to the top of your YAML file

You will then get auto completion and validation, for example, at the start of a line type kafka at the start of the line to trigger the default snippets (templates) for Kafka.

For Schema registry, type Schema, Connect type Connect, Alerting type altering, etc.

You require at minimum a lenses-hq connection and a kafka connection for the schema to be valid.

You do not need to use the default snippets, you can also use the auto completion for each connection type.

Provisioning

This page describes how to setup connections to Kafka and other services and have changes applied automatically for the Lenses Agent.

HQ

This page describes connection a Lenses Agent with HQ

To be able to view and drill into your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

Only one HQ connection is allowed.

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "\${ENV_VAR_NAME}"

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: "\${LENSES_HQ_HOST}"
      port:
        value: 10000
      agentKey:
        value: "\${LENSES_HQ_AGENT_KEY}"
      sslEnabled:
        value: true
      sslTruststore:
        file: hq-truststore.jks
      sslTruststorePassword:
        value: "\${LENSES_HQ_AGENT_TRUSTSTORE_PWD}"

Kafka

This page describes how to connect the Lenses Agent to your Kafka brokers.

The Lenses Agent can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

For JMX see

Aiven

This page describes configuring Lenses to connect to Aiven.

Only one Kafka connection is allowed.

The name must be kafka.

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "\${ENV_VAR_NAME}"

Find your Service URI

From the Aiven, locate your Service URI and set it as the bootstrap servers.

Configure Provisioning

Set the following in the provisioning.yaml, replacing Service URI, username and password from your Aiven account.

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[Service URI]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="[your-username]"
        password="[your-password]";

AWS MSK

This page describes connection the Lenses Agent to a AWS MSK cluster.

Only one Kafka connection is allowed.

The name must be kafka.

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "\${ENV_VAR_NAME}"

It is recommended to install the Agent on an EC2 instance or with EKS in the same VPC as your MSK cluster. The Agent can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Agent installation.

Enable Open Monitoring

If you want to have the Agent collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

It is not recommended to use Plaintext for secure environments. For these environments use TLS or IAM.

When the Agent is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

Configure Provisioning

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsType:
      value: AWS

Azure HDInsight

This page describes connection Lenses to a Azure HDInsight cluster.

Only one Kafka connection is allowed.

The name must be kafka.

See for support.

Environment variables are supported; escape the dollar sign

Find your Kafka endpoints

In our Azure Portal, go to Dashboards->Ambari home. Go to Kafka->Configs->Kafka Broker->Kafka Broker hosts

Optionally find your Zookeeper endpoints

Optionally get the Zookeeper endpoints: Go to Zookeeper->Configs->Zookeeper Server->Zookeeper Server hosts.

Configure Provisioning

See for different configurations for your security protocols.

Environment variables are supported; escape the dollar sign

Confluent Cloud

This page describes configuring Lenses to connect to Confluent Cloud.

For Confluent Platform see Apache Kafka.

Only one Kafka connection is allowed.

The name must be kafka.

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "\${ENV_VAR_NAME}"

Create a Data Integration API key

From Data integration API keys, select Create Key.
For this guide select Global access to get your API Key and API Secret Key.
Go to Cluster Settings to get your Bootstrap Server.

Configure Provisioning

Set the following in the provisioning.yaml

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "\${ENV_VAR_NAME}"

provisioning.yaml

kafka:
- name: kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required 
        username="[YOUR_API_KEY]" 
        password="[YOUR_API_KEY_SECRET]";

Confluent Platform

This page describes configuring Lenses to connect to Confluent Platform.

For Confluent Platform see

IBM Event Streams

This page describes how to connect Lenses to IBM Event Streams.

IBM Event Streams requires a replication factor of 3. Ensure you set the replication factor accordingly for Lenses internal topics.

See .

Only one Kafka connection is allowed.

The name must be kafka.

See for support.

Environment variables are supported; escape the dollar sign

Find your bootstrap endpoints

From the IBM Cloud console, locate your bootstrap_endpoints, for the service credentials you want to connect with.

Configure Provisioning

Set the following in the provisioning.yaml:

Use "token" as the username in the Jaas Config. Set the password as your API KEY from IBM Event streams

Environment variables are supported; escape the dollar sign

Schema Registries

This page describes adding a Schema Registries to the Lenses Agent.

Overview

This page describes an overview of connecting a Lenses Agent with Schema Registries

Consider if you have a high number of schemas.

Only one Schema Registry connection is allowed.

Authentication

TLS and basic authentication are supported for connections to Schema Registries.

JMX Metrics

The Agent can collect Schema registry metrics via:

JMX
Jolokia

See

Supported formats

AVRO
PROTOBUF

JSON and XML formats are supported by Lenses but without a backing schema registry.

Schema deletion

To enable the deletion of schemas in the UI, set the following in the lenses.conf file.

IBM Event Streams supports hard deletes only

Apicurio

This page describes connecting Lenses to Apicurio.

Apicuro supports the following versions of Confluent's API:

Confluent Schema Registry API v6
Confluent Schema Registry API v7

Only one Schema Registry connection is allowed.

Name must be schema-registry.

See for support.

Environment variables are supported; escape the dollar sign

Set the schema registry URLs to include the compatibility endpoints, for example:

IBM Event Streams Registry

This page describes connecting Lenses to IBM Event Streams schema registry.

Requires Enterprise subscription on IBM Event Streams and only hard delete is supported for IBM Event streams

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

Use "token" as the username. Set the password as your API KEY from IBM Event streams

Only one Schema Registry connection is allowed.

Name must be schema-registry.

See for support.

Environment variables are supported; escape the dollar sign

Hardware & OS

This page describes the hardware and OS prerequisites for Lenses.

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

TLS

This page describes how to configure TLS for the Lenses Agent.

By default, the Agent does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Agent or by using a TLS proxy or load balancer.

Global Truststore

To use a non-default global truststore, set the path in accordingly with the LENSES_OPTS variable.

Custom Truststore

Mutual TLS

To enable mutual TLS, set your keystore accordingly.

Rate Limiting

Rate limit the calls the Lenses Agent makes to Schema Registries and Connect Clusters.

To rate limit the calls the Agent makes to Schema Registries or Connect Clusters set the following the Agent configuration:

The exact values provided will depend on your setup, for example the number of schemas, how often are new schemas added, so some trial and error is required.

JVM Options

This page describes the JVM options for the Lenses Agent.

The Agent runs as a JVM app; you can tune runtime configurations via environment variables.

Key

Description

LENSES_OPTS

For generic settings, such as the global truststore. Note that the docker image is using this to plug in a prometheus java agent for monitoring Lenses

LENSES_HEAP_OPTS

JVM heap options. Default setting are -Xmx3g -Xms512m that sets the heap size between 512MB and 3GB. The upper limit is set to 1.2GB on the Box development docker image.

LENSES_JMX_OPTS

Tune the JMX options for the JVM i.e. to allowing remote access.

LENSES_LOG4J_OPTS

Override Agent logging configuration. Should only be used to set the logback configuration file, using the format -Dlogback.configurationFile=file:/path/to/logback.xml.

LENSES_PERFORMANCE_OPTS

JVM performance tuning. The default settings are -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=

Logs

This page describes configuring Lenses Agent logging.

Changes to the logback.xml are hot reloaded by the Agent, no need to restart.

All logs are emitted unbuffered as a stream of events to both stdout and to rotating files inside the directory logs/.

The logback.xml file is used to configure logging.

If customization is required, it is recommended to adapt the default configuration rather than write your own from scratch.

The file can be placed in any of the following directories:

the directory where the Agent is started from
/etc/lenses/
agent installation directory.

The first one found, in the above order, is used, but to override this and use a custom location, set the following environment variable:

export LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/path/to/logback.xml"

Default configuration

The default configuration file is set up to hot-reload any changes every 30 seconds.

Log Level

The default log level is set to INFO (apart from some very verbose classes).

Log Format

All the log entries are written to the output using the following pattern:

%d{ISO8601} %-5p [%c{2}:%L] [%thread] %m%n

You can adjust this inside logback.xml to match your organization’s defaults.

Log Location

logs/ you will find three files: lenses.log, lenses-warn.log and metrics.log. The first contains all logs and is the same as the stdout. The second contains only messages at level WARN and above. The third one contains timing metrics and can be useful for debugging.

Log Buffering

The default configuration contains two cyclic buffer appenders: "CYCLIC-INFO” and “CYCLIC-METRICS”. These appenders are required to expose the Agent logs within the Admin UI.

Upgrade to Lenses 6.1

There's no breaking change between 6.0 and 6.1.

User Guide

Environments

This page describes Environments in Lenses.

Environments are virtual containers for you, including Kafka Cluster, Schema Registries, and Kafka Connect Clusters.

Each Environment has an Agent, the Agent communicates with HQ via an Agent Key generated at the environment creation time.

Environments can be assigned tiers, domains and a description and group accordingly.

Go to Environments in the left-hand side navigation, then select New Environments button in the top right corner.

Enter the details for the agent, once you have a key you will be guided on how to run and start docker, then configure the agent to connect to your environment.

Learn how to configure an agent here.

Identity & Access Management

This page describes Identity & Access Management (IAM) in Lenses.

Topics

This page describes exploring topics in Lenses.

Overview

This page describes an overview of Lenses IAM (Identify & Access Management)

Principals (Users & Service accounts) receive their permissions based on their group membership.

Roles hold a set of policies, defining the permissions. Roles are assigned to groups.

Roles provide flexibility in how you want to provide access. You can create a very open policy or a very granular policy, for example, allowing operators and support engineers certain permissions to restart Connectors but denying actions that would allow them to view data or configuration options.

Roles are defined at the HQ level. This allows you to control access to actions at HQ and lower environment levels and to assign the same set of permissions across your whole Kafka landscape in a central place.

Policies

A policies have:

One or more actions;
One or more resource patterns that the actions apply to;
An effect: allow or deny.

name: [policy_name]
policy: #list of actions/resources/effect
- action:
  resource:
  effect:[allow|deny]

If any effect is deny for a resource the result is always deny, the principle of least privileged applies.

A policy is defined by a YAML specification.

Example Policy

name: blue-things
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
      - environments:AccessEnvironment
    resource: environments:*
    effect: allow
  - action:
      - kafka:*
      - schemas:*
      - kafka-connect:*
      - kubernetes:*
      - applications:*
    resource:
      - kafka:topic:*/*/blue-*
      - kafka:consumer-group:*/*/blue-*
      - kafka:acl:*/*/*/user/blue-*
      - schemas:schema:*/*/blue-*
      - kafka-connect:cluster:*/*
      - kafka-connect:connector:*/*/blue-*
      - sql-streaming:sql-processor:*/*/*/blue-*
      - kubernetes:cluster:*/*
      - kubernetes:namespace:*/*/*
      - applications:external-application:*/blue-*

Actions

Actions describe a set of actions. Concrete actions can match an Action Pattern. In this text, action and action patterns are used interchangeably.

An action has the format: service:operation, e.g. iam:DeleteUser

Services

Services describe the system entity that an action applies to. Services are:

environments
kafka
registry
schemas
kafka-connect
sql-streaming
kubernetes
applications
alerts
data-policies
governance
audit
iam
administration

Operations

Operation can contain a wildcard. If so, only at the end. See IAM Reference for the available operations per service.

Resources

Resources identify which resource, in a service, that the principal is allowed or denied, to perform the operation on.

Resource-type cannot contain a combination of characters with wildcards.

If the service is provided, resource-type can be a wildcard.

Resource ID

The resource ID identifies the resource within the context of a service and a resource type.

A resource-id consists of one or more segments separated by a slash /. A segment can be a wildcard, or contain a wildcard as a suffix of a string. If a segment is a wildcard, then remaining segments do not need to be provided, and will be assumed to be wildcards as well.

The format is service:resource-type:resource-id

Where LRN is the Lenses Resource Name

kafka:topic:my-env/* will be expanded to kafka:topic:my-env/*/*;
kafka:topic:my-env/my-cluster* is invalid because the Topic segment is missing, kafka:topic:my-env/my-cluster*/topic would be valid though;
*:topic:* is invalid, the service is not provided;
kaf*:* and kafka:top* are invalid, service and resource-type cannot contain wildcards;
kafka:*:foo is invalid, if the resource-type is a wildcard then resource-id cannot be set.

Evaluation

A principal (user or service account) can perform an action on a resource if:

In any of the roles it receives via group membership:

There is any matching Permission Statement that has an effect of allow;
And there is not any matching Permission Statement that has an effect of deny.

A Permission Statement matches an action plus resource, if:

The action matches any of the Permission Statement's Action Patterns, AND:
The resource matches any of the Permission Statement's Resource Patterns.

An Action matches an Action Pattern (AP) if:

The AP is a wildcard, OR:
The Action's service equals the AP's and the AP's operation string-matches the Action's operation.

A Resource matches a Resource Pattern (RP) if:

The RP is a wildcard, OR:
The Resource's services equals the RP's and the RP's resource-type is a wildcard, OR:
The Resource's service and types equals that of the RP and resource-ids match. Resource-ids are matched by string-matching each individual segment. If the RP has a trailing wildcard segment, the remaining segments are ignored.

A string s matches p if:

They equal character by character.
If s or p has more non-wildcard characters than the other they don't match;
If p contains a * suffix, any remaining characters in s are ignored.

match

"lit"

true

"lit"

"li"

false

"lit"

"litt"

false

"lit"

"oth"

false

"*"

"some"

true

"foo*"

"foo"

true

"foo*"

"foo-bar"

true

"x"

false

"x"

false

Order of items in any collection is irrelevant during evaluation. Collections are considered sets rather than ordered lists. The following are equivalent:

Order of Resource Patterns does not matter
Order of Permission Statements does not matter
Order of Roles does not matter
Order of Groups does not matter

Examples

In the examples we're not too religious about strict JSON formatting.

Broad Allow + Specific Deny

Given:

policy:
  - effect: allow
    resource: kafka:topic:my-env/*/*
    action: ReadKafkaData
  - effect: deny
    resource: kafka:topic:*/*/forbidden-topic
    action: ReadKafkaData

A principal:

Can ReadKafkaData on kafka:topic:my-env/the-cluster/some-topic because it is allowed and not denied;
Cannot DeleteKafkaTopic on kafka:topic:my-env/the-cluster/some-topic because there is no allow;
Cannot ReadKafkaData on kafka:topic:my-env/the-cluster/forbidden-topic because while it is allowed the deny kicks in.

Multiple Resources I

Given:

policy:  
  - effect: allow
    resource: [*, kafka:topic:my-cluster/*]
    action: ReadKafkaData

A principal:

Can ReadKafkaData on kafka:topic:someone-else-cluster/their-topic because the resource matches *.

Note that here the matching can be considered "most permissive".

Multiple Resources II

Given:

policy:
  - effect: allow
    resource: [kafka:topic:my-cluster/my-topic-1, kafka:topic:my-cluster/my-topic-2]
    action: ReadKafkaData

A principal:

Can ReadKafkaData on kafka:topic:my-cluster/my-topic-1 and kafka:topic:my-cluster/my-topic-2 because the resources match, but cannot ReadKafkaData on kafka:topic:my-cluster/my-topic-3.

Deploying an Agent

This page describes installing Lenses Agent in Kubernetes via Helm.

Lenses HQ must be installed before setting up an Agent.

Latest Agent container image is here on Docker Hub.

Helm Charts available here.

Run the following commands to add the charts to your Helm repo.

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

Prerequisites

Kubernetes 1.23+
Helm 3.8.0+
Available local Postgres database instance.
- If you need to install Postgres on Kubernetes you can use one of many different publicly available Helm charts such as Bitnami's.
- Or you can use a cloud provider's Postgres service such as one of these: AWS, Azure, or GCP.
- See Lenses docs here for information on configuring Postgres to work with Agent.
Follow these steps to configure your Postgres database for Lenses Agent.
External Secrets Operator is the only supported secrets operator.

Configure an Agent

In order to configure an Agent, we have to understand the parameter groups that the Helm Chart offers.

Under the lensesAgent parameter there are some key parameter groups that are used to set up the connection to Lenses HQ:

Storage
HQ connection
Provision
Cluster RBACs

Moving forward, in the same order you can start configuring your Helm chart.

JSON Schema Support

You can use JSON schema support to help you configure the Values files for Helm. See JSON schema for support. Included in the repository is a JSON schema for the Agent Helm chart.

Configuring Agent chart

Configure storage (Postgres / H2 - Embedded database)

Postgres database is recommended for Production and Non-production workloads.

H2 embedded database is recommended to Evaluation purposes.

Running Agent with Postgres database

Prerequisite:

Running Postgres instance;
Created database for an Agent;
Username (and password) which has access to the created database;

In order to successfully run HQ, storage is within values.yaml has to be defined first.

The definition of storage object is as follows:

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: ""
      port: 
      username: ""
      database: ""
      schema: ""
      params: {}

Alongside Postgres password, which can be referenced/created through Helm chart, there are a few more options which can help while setting up HQ.

There are two ways how username can be defined:

The most straightforward way, if the username is not being changed, is by just defining it within the username parameter such as

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: lenses

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: external  # use "external" to manage it using secrets
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_USERNAME
      valueFrom:
        secretKeyRef:
          name: [SECRET_RESOURCE_NAME]
          key: [SECRET_RESOURCE_KEY]

Password reference types

Postgres password can be handled in three ways using:

Pre-created secret;
Creating secrets on the spot through values.yaml;

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.playground.svc.cluster.local
      port: 5432
      username: lenses
      database: lensesagent
      password: useOnlyForDemos

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: lenses
      password: external   # use "external" to manage it using secrets
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: [SECRET_RESOURCE_NAME]
          key: [SECRET_RESOURCE_KEY]

Running Agent with H2 embedded database

Embedded database is not recommended to be used in Production or high load environments.

In order to run Agent with H2 embedded database there are few things to be aware about:

K8s cluster Agent will be deployed on has to support Persistent Volumes;
Postgres options in Helm chart has to be left out.

values.yaml

persistence:
  storageH2:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 300Mi

Configure HQ connection (agent key)

Connection to Lenses HQ is a straight forward process which requires two steps:

Creating Environment and obtaining AGENT KEY in HQ as described here, if you already have not done so;
Storing that same key in Vault or as a K8s secret.

The agent communicates with HQ via a secure custom binary protocol channel. To establish this channel and authenticate the Agent needs and AGENT KEY.

Once the AGENT KEY has been copied, store it inside of Vault or any other tool that has integration with Kubernetes secrets.

There are three available options how the agent key can be used:

ExternalSecret via External Secret Operator (ESO)
Pre-created secret
Inline string

To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying Agent.

When specifying secret.type: "externalSecret", the chart will:

create an ExternalSecret in the namespace where Agent is deployed;
a secret is mounted for Agent to use.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "externalSecret"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key
        externalSecret:
          additionalSpecs: {}
          secretStoreRef:
            type: ClusterSecretStore # ClusterSecretStore | SecretStore
            name: [secretstore_name]

Make sure that secret you are going to use is already created in namespace where Agent will be installed.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "precreated"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key

This option is NOT for PRODUCTION usage but rather just for demo / testing.

The chart will create a secret with defined values below and the same secret will be read by Agent to connect to HQ.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        # Secret name where agentKey will be read from
        name: "lenses-agent-secret-1"
        # Value of agentKey generated by HQ
        value: "agent_key_*"

This secret will be fed into the provisioning.yaml. The HQ connection is specified below, where reference ${LENSESHQ_AGENT_KEY} is being set:

values.yaml

lensesAgent:
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [LENSES_HQ_FQDN_OR_IP]
            port:
              value: 10000
            agentKey:
              # This property shouldn't be changed as it is mounted automatically
              # based on secret choice for hq.agentKey above 
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false

In order to enable TLS for secure communication between HQ and the Agent please refer to the following part of the page.

Configure provisioning (Kafka / SchemaRegistry / Kafka Connect)

Provisioning offers various connections starting with:

Kafka ecosystem components such as:
Alerts & Audits
AWS

values.yaml

lensesAgent:
  provision:
    path: /mnt/provision-secrets
    connections:
      # Kafka Connection
      kafka:
        - name: Kafka
          version: 1
          tags: [my-tag]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://your.kafka.broker.0:9092
                - PLAINTEXT://your.kafka.broker.1:9092
            protocol: 
              value: PLAINTEXT
            # all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Confluent Schema Registry Connection
      confluentSchemaRegistry:
        - name: schema-registry
          tags: ["tag1"]
          version: 1      
          configuration:
            schemaRegistryUrls:
              value:
                - http://my-sr.host1:8081
                - http://my-sr.host2:8081
            ## all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Kafka Connect connection
      connect:
        - name: my-connect-cluster-name
          version: 1    
          tags: ["tag1"]
          configuration:
            workers:
              value:
                - http://my-kc.worker1:8083
                - http://my-kc.worker2:8083
            metricsPort: 
              value: 9585
            metricsType: 
              value: JMX

values.yaml

lensesAgent:
  additionalEnv:
    - name: SASL_JAAS_CONFIG
      valueFrom:
        secretKeyRef:
          name: kafka-sharedkey
          key: sasljaasconfig
  provision:
    path: /mnt/provision-secrets
    connections:
      # Kafka Connection
      kafka:
        - name: kafka
          version: 1
          tags: [ "dev", "dev-2", "eu"]
          configuration:
            kafkaBootstrapServers:
              value:
                - SASL_SSL://test-dev-2-kafka-bootstrap.kafka-dev.svc.cluster.local:9093
            saslJaasConfig:
              value: ${SASL_JAAS_CONFIG}
            saslMechanism:
              value: SCRAM-SHA-512
            protocol:
              value: SASL_SSL
      # Confluent Schema Registry Connection
      confluentSchemaRegistry:
        - name: schema-registry
          tags: ["tag1"]
          version: 1      
          configuration:
            schemaRegistryUrls:
              value:
                - http://my-sr.host1:8081
                - http://my-sr.host2:8081
            ## all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Kafka Connect connection
      connect:
        - name: my-connect-cluster-name
          version: 1    
          tags: ["tag1"]
          configuration:
            workers:
              value:
                - http://my-kc.worker1:8083
                - http://my-kc.worker2:8083
            metricsPort: 
              value: 9585
            metricsType: 
              value: JMX

More about provisioning and more advanced configuration options for each of these components can be found on the following link.

Cluster RBACs

The Helm chart creates Cluster roles and bindings, that are used by SQL Processors, if the deployment mode is set to KUBERNETES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.

To disable the creation of Kubernetes RBAC set: rbacEnabled: false

If you want to limit the permissions the Agent has against your Kubernetes cluster, you can use Role/RoleBinging resources instead. Follow this link in order to enable it.

If you are not using SQL Processors and want to limit permissions given to Agent's ServiceAccount, there are two options you can choose from:

rbacEnable: true - will enable the creation of ClusterRole and ClusterRoleBinding for service account mentioned above;

values.yaml

rbacsEnable: true
namespaceScope: false

rbacEnable: true and namespaceScope: true - will enable the creation of Role and RoleBinding which is more restrictive;

values.yaml

rbacsEnable: true
namespaceScope: true

(Optional) Enable TLS connection with HQ

In this case, TLS has to be enabled on HQ. In case you haven't yet enabled it, you can find details here to do it.

Enabling TLS in communication between HQ is being done in the provisioning part of values.yaml.

In order to successfully enable TLS for the Agent you would need to:

additionalVolume & additionalVolumeMounts - with which you will mount truststore with CA certificate that HQ is using and which Agent will need to successfully pass the handshake.
additionalEnv - which will be used to securely read passwords to unlock truststore.
Enable SSL in provision.

values.yaml

# Additional Volume with CA that HQ uses
additionalVolumes:
  - name: hq-truststore
    secret:
      secretName: hq-agent-test-authority
additionalVolumeMounts:
  - name: hq-truststore
    mountPath: "/mnt/provision-secrets/hq"

lensesAgent:
 # Additional Env to read truststore password from secret
 additionalEnv:
    - name: LENSES_HQ_AGENT_TRUSTSTORE_PWD
      valueFrom:
        secretKeyRef:
          name: hq-agent-test-authority
          key: truststore.jks.password
 provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: true
            sslTruststore:
              file: "/mnt/provision-secrets/gq/truststore.jks"
            sslTruststorePassword:
              value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

(Optional) Services

Enable a service resource in the values.yaml:

# Lenses service
service:
  enabled: true
  annotations: {}

(Optional) Controlling resources

To control the resources used by the Agent:

# Resource management
resources:
  requests:
    cpu: 1
    memory: 4Gi
  limits:
    cpu: 2
    memory: 5Gi

In case LENSES_HEAP_OPTS is not set explicitly it will be set implicitly.

Examples:

if no requests or limits are defined, LENSES_HEAP_OPTS will be set as -Xms1G -Xmx3G
If requests and limits are defined above defined values, LENSES_HEAP_OPTS will be set by formula -Xms[-Xmx / 2] -Xmx[limits.memory - 2]
If .Values.lenses.jvm.heapOpts it will override everything

Enabling SQL processors in K8s mode

To enable SQL processor in KUBERENTES mode and control the defaults:

lensesAgent:
  sql:
    processorImage: hub.docker.com/r/lensesioextra/sql-processor/
    processorImageTag: latest
    mode: KUBERNETES
    heap: 1024M
    minHeap: 128M
    memLimit: 1152M
    memRequest: 128M
    livenessInitialDelay: 60 seconds

To control the namespace Lenses can deploy processors, use the sql.namespaces value.

SQL Processor Role Binding

To achieve you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to.

For example:

Lenses namespace = lenses-ns
Processor namespace = lenses-proc-ns

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role
  namespace: lenses-proc-ns
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role-binding
  namespace: lenses-proc-ns
subjects:
- kind: ServiceAccount
  namespace: lenses-ns
  name: default
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io

Finally you need to define in the Agent configuration which namespaces the Agent has access to. Amend values.yaml to contain the following:

values.yaml

lensesAgent:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "lenses-processors"
        ]
      }

Persistence Volume

Persistence can be enabled for three purposes:

Use H2 embedded database
Logging
Provisioning

When using the Data Policies module to persist your data policies rules
When lenses.storage.enabled: false and an H2 local filesystem database is used instead of PostgreSQL
For non critical and NON PROD deployments

Configuration:

values.yaml

persistence:
  storageH2:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 20Gi
    storageClass: ""
    annotations: {}
    existingClaim: ""

When you need persistent log storage across pod restarts
When you want to retain logs for auditing or debugging purposes

Configuration:

values.yaml

persistence:
  log:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 5Gi
    storageClass: ""
    annotations: {}
    existingClaim: ""

Dedicated volume for provisioning data managed via the HQ.

When to enable:

When using HQ-based provisioning workflows
Must be combined with PROVISION_HQ_URL and PROVISION_AGENT_KEY environment variables

Configuration:

values.yaml

persistence:
  provisioning:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 5Mi
    storageClass: ""
    annotations: {}
    existingClaim: ""

or Helm command execution:

# Install the Chart.
helm repo add lensesio https://helm.repo.lenses.io/
helm repo update
# Deploy the Agent. Only available from version 6.1.0 onwards.
helm install lenses-agent \
  lensesio/lenses-agent \
  --set 'persistence.provisioning.enabled=true' \
  --set 'lensesAgent.additionalEnv[0].name=PROVISION_HQ_URL' \
  --set 'lensesAgent.additionalEnv[0].value=[lenses-hq.url]' \
  --set 'lensesAgent.additionalEnv[1].name=PROVISION_AGENT_KEY' \
  --set 'lensesAgent.additionalEnv[1].value=[agent_key_*]'

Prometheus metrics

Prometheus metrics are automatically exposed on port 9102 under /metrics.

At this very moment you can scrape it only via Service under port called http-metrics.

lenses.conf

The main configurable options for lenses.conf are available in the values.yaml under the lenses object. These include:

Authentication
Database connections
SQL processor configurations

To apply other static configurations use lenses.append.conf, for example:

values.yaml

lensesAgent:
  append:
    conf: |
      lenses.interval.user.session.refresh=40000

Install the Chart

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

Installing the Agent

Installing using cloned repository:

helm install lenses-agent charts/lenses-agent \
   --values charts/lenses-agent/values.yaml \
   --create-namespace --create lenses-agent

Installing using Helm repository:

terminal

helm install lenses-agent lensesio/lenses-agent \
   --values values.yaml \
   --create-namespace --namespace lenses-agent \
   --version 6.0.0

Be aware that for the time being and alpha purposes usage of --version is mandatory when deploying Helm chart through Helm repository.

Example Values files

Be aware that example of values.yaml shows only how all parameters should look at the end. Please fill them with correct values otherwise Helm installation might not be successful.

Example of values.yaml

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: [postgres.url]
      port: 5432
      username: postgres 
      password: changeMe
      database: agent
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: "agentKey"
        value: "agent_key_*"
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: hq-tls-test-lenses-hq.hq-agent-test.svc.cluster.local
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
      kafka:
        # There can only be one Kafka cluster at a time
        - name: kafka
          version: 1
          tags: ['staging', 'pseudo-data-only']
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://kafka-1.svc.cluster.local:9092
                - PLAINTEXT://kafka-2.svc.cluster.local:9092
            protocol:
              value: PLAINTEXT
            # Metrics are strongly suggested for better Kafka cluster observability
            metricsType:
              value: JMX
            metricsPort:
              value: 9581

You can also find examples in the Helm chart repo.

Deploying HQ

This page describes installing Lenses HQ in Kubernetes via Helm.

Lenses HQ is prerequisite for installation of Lenses Agent

Latest images:

Latest HQ image available here from Docker Hub.

Latest Lenses CLI as tarball or as a container.

Prerequisites

Kubernetes 1.23+
Helm 3.8.0+
Available local Postgres database instance:
- If you need to install Postgres on Kubernetes you can use one of the many publicly available Helm charts such as Bitnami's.
- Or you can use one of the cloud provider's Postgres services such as one of these: AWS, Azure, or GCP.
- See Lenses docs here for information on configuring Postgres to work with HQ .
- username (and password) that has access to HQ database;
External Secrets Operator is the only supported secrets operator.

Configure HQ

To configure Lenses HQ properly we have to understand the parameter groups that the Chart offers.

Under the lensesHq parameter there are some key parameter groups that are used to set up HQ:

storage
- definition of connection towards database (Postgres is the only storage option)
auth
- Password based authentication configuration
- SAML / SSO configuration
- definition of administrators or first users to access the HQ
http
- defines port under which HQ will be available for end users
- defines values of special headers and cookies
- types of connection such as TLS and non-TLS definitions
agents
- defines connection between HQ and the Agent such as port where HQ will be listening for agent connections.
- types of connection such as TLS and non-TLS definitions
license
monitoring
- controls the metrics settings where Prometheus alike metrics will be exposed
loggers
- definition of logging level for HQ

Moving forward, in the same order you can start configuring your Helm chart.

Configure storage (Postgres)

Postgres is the only available storage option.

Prerequisite:

Running Postgres instance;
Created database for HQ;
Username (and password) which has access to the created database;

In order to successfully run HQ, storage within values.yaml has to be defined first.

Definition of storage object is as follows:

lensesHq:
  storage:
    postgres:
      enabled: true
      host: ""
      port: 
      username: ""
      database: ""
      schema: ""
      tls: 
      params: {}
      passwordSecret:
        type: ""

Alongside Postgres password, which can be referenced / created through Helm chart, there are few more options which can help while setting up HQ.

Username reference types

There are two ways how username can be defined:

The most straightforward way, if the username is not being changed, is by just defining it within the username parameter such as

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses

In case Postgres username is being rotated or frequently changed it can be referenced from pre-created secret

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses
      useSecretForUsername:
        enabled: true
        existingSecret:
          name: my-secret
          key: username

Password reference types

Postgres password can be handled in three ways using:

External Secret via ExternalSecretOperator;
Pre-created secret;
Creating secret on the spot through values.yaml;

To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying HQ.

When specifying passwordSecret.type: "externalSecret", the chart will:

create an ExternalSecret in the namespace where HQ is deployed;
a secret is mounted for HQ to use.

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.playground.svc.cluster.local
      port: 5432
      username: lenses
      database: lenseshq
      passwordSecret:
        type: "externalSecret"
        # Secret name where database password will be read from
        name: hq-password
        # Key name under secret where database password is stored
        key: password
        externalSecret:
          additionalSpecs: {}
          secretStoreRef:
            type: SecretStore # or ClusterSecretStore
            name: secretstore-secrets

Make sure that secret you are going to use is already created in namespace where HQ will be installed.

values.yaml

 lensesHq:
   storage:  
     postgres:
       enabled: true
       host: postgres-postgresql.playground.svc.cluster.local
       port: 5432
       username: lenses
       database: lenseshq
       passwordSecret:
         type: "precreated"
         # Secret name where database password will be read from
         name: hq-password
         # Key from secret's data where database password is being stored
         key: postgres-password

This option is NOT for PRODUCTION usage but rather just for demo / testing.

The chart will create a secret with defined values below and the same secret will be read by HQ in order to connect to Postgres.

values.yaml

 lensesHq:
   storage:
     postgres:
       enabled: true
       host: [POSTGRES_HOSTNAME]
       port: 5432
       username: lenses
       database: lenseshq
       passwordSecret:
         type: "createNew"
         # name of a secret that will be created
         name: [K8s_SECRET_NAME]
         # Database password
         password: [DATABASE_USER_PASSWORD]

Advanced Postgres settings

Sometimes to form the correct connection URI special parameters are needed. You can set the extra settings using params.

Example:

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses
      params:
        sslmode: require

Configure AUTH endpoint

SAML / SSO is available only with Enterprise license.

The second pre-requirement to successfully run HQ is setting initial authentication.

You can choose between:

password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.

Assertion Consumer Service endpoint is following

/api/v2/auth/saml/callback?client_name=SAML2Client

The definition of auth object is as follows:

values.yaml

lensesHq:
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG
    administrators:
      - admin
      - [email protected]
      - [email protected]
    saml:
      enabled: true
      baseURL: ""
      entityID: ""
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: ""
        secretKeyName: ""
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          </md:EntityDescriptor>
  
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
      uiRootURL: "/"
      groupAttributeKey: "groups"
      authnRequestSignature:
        enabled: false

First to cover is the users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. Passwords must be hashed using bcrypt before being placed within the password property, for security purposes, ensuring that they are stored correctly and securely.

values.yaml

lensesHq:
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG

values.yaml

lensesHq:
  auth:
    users:
      - username: $(ADMIN_USER)
        password: $(ADMIN_USER_PWD)
  additionalEnv:
    - name: ADMIN_USER
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-username
    - name: ADMIN_USER_PWD
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-password

Second, to cover will be administrators. It serves as the definition of user emails have the highest level of permissions upon authentication to HQ.

Third attribute is saml.metadata field, needed for setting SAML / SSO authentication. In this step, you will need metadata.xml file which can be set in two ways:

Referencing metadata.xml file through pre-created secret;
Placing metadata.xml contents inline as a string.

lensesHq:
  auth:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    administrators:
      - [email protected]
      - [email protected]
    saml:
      baseURL: ""
      entityID: ""
      metadata:
        referenceFromSecret: true
        secretName: hq-tls-mock-saml-metadata
        secretKeyName: metadata.xml
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "manual"

lensesHq:
  auth:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    administrators:
      - [email protected]
      - [email protected]
    saml:
      baseURL: ""
      entityID: ""
      metadata:
        referenceFromSecret: false
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          ...
          ...
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"

In case SAML IdP requires certificate verification, same can be enabled and provided in the following way:

values.yaml

lensesHq:
  auth:
    saml:
      authnRequestSignature:
        enabled: true
        authnRequestSigningCert:
          referenceFromSecret: true
          secretName: hq-agent-test-authority
          secretKeyName: hq-tls-test.crt.pem
        authnRequestSigningKey:
          secret:
            name: saml-test
            key: privatekey.key

values.yaml

lensesHq:
  auth:
    saml:
      authnRequestSignature:
        enabled: true
        authnRequestSigningCert:
          stringData: |
            -----BEGIN CERTIFICATE-----
            ....
            -----END CERTIFICATE-----
        authnRequestSigningKey:
          secret:
            name: saml-test
            key: privatekey.key

Configure HTTP endpoint

The third pre-requirement to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around the HTTP endpoint of the HQ itself and how users will interact with it.

Definition of HTTP object is as follows:

lensesHq:
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    accessControlAllowCredentials: false
    secureSessionCookies: true
    tls:
      enabled: true
      cert:
      privateKey:
        secret:
          name: 
          key:

Second part of HTTP definition would be enabling TLS and TLS definition itself. As previously defined for lensesHq.agents.tls same way of configuring TLS can be used for lensesHq.http.tls definition as well.

Configure agent's connection endpoint

After correctly configuring the authentication strategy and connection endpoint, the agent handling is the last most important box to tick.

The Agent's object is defined as follows:

lensesHq:
  agents:
    # which port to listen on for agent requests
    address: ":10000" 
    tls:
      enabled: false
      verboseLogs: false
      cert:
      privateKey:

Enabling TLS

By default TLS for the communication between Agent and HQ is disabled. In case the requirement is to enable it, fthe ollowing has to be set:

lensesHq.agents.tls - certificates to manage the connection between HQ and the Agents
lensesHq.http.tls- certificates to manage connection with HQ's API

Unlike private keys which can be referenced and obtained only through a secret, Certificates can be referenced directly in values.yaml file as a string or as a secret.

values.yaml

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        referenceFromSecret: true
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem

values.yaml

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        stringData: |
          -----BEGIN CERTIFICATE-----
          ...
          ...
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem

Configure license

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

License can be read in multiple ways:

from a pre-created secret
directly as a string defined in values.yaml file

values.yaml

lensesHq:
  license:
    referenceFromSecret: true
    secretName: hq-license
    secretKeyName: key
    acceptEULA: true

values.yaml

lensesHq:
  licence:
    referenceFromSecret: false
    stringData: "license_key_*"
    acceptEULA: true

Configure metrics endpoint

Metrics are optionally available in a Prometheus format and by default served on port 9090.

The port can be changed in the following way:

values.yaml

lensesHq:
  metrics:
    prometheusAddress: ":9090"

(Optional) Configure Ingress & Services

Whilst the chart supports setting TLS on Lenses HQ itself we recommend placing it on the Ingress resource

Ingress and service resources are optionally supported.

The http ingress is intended only for HTTP/S traffic, while the agents ingress is designed specifically for TCP protocol. Ensure appropriate ingress configuration for your use case.

Enable an Ingress resource in the values.yaml:

values.yaml

ingress:
  http:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
    host: example.com
    ingressClassName: ""
    tls:
      enabled: false
      # The TLS secret must contain keys named tls.crt and tls.key that contain the certificate and private key to use for TLS.
      secretName: ""


  agent:
    enabled: true
    agentIngressConfig:
      apiVersion: traefik.containo.us/v1alpha1
      kind: IngressRouteTCP
      metadata:
        name: agents
      spec:
        entryPoints:
          - agents
        routes:
          - match: HostSNI(`example.com`)  # HostSNI to match TLS for TCP
            services:
              - name: lenses-hq            # Replace with your service name
                port: 10000                # Agent default TCP port  
        tls: {}

Enable a service resource in the values.yaml:

values.yaml

# Lenses HQ service
service:
  enabled: true
  type: ClusterIP
  annotations: {}
  externalTrafficPolicy:
  loadBalancerIP: 130.211.x.x
  loadBalancerSourceRanges:
    - 0.0.0.0/0

(Optional) Configure Service Accounts

Lenses HQ, by default, uses the default Kubernetes service account but you can choose to use a specific one.

If the user defines the following:

values.yaml

# serviceAccount is the Service account to be used by Lenses to deploy apps
serviceAccount:
  create: true
  annotations: {}
  name: lenses-hq

The chart will create a new service account in the defined namespace for HQ to use.

(Optional) Enable RBAC

There are two options you can choose between:

rbacEnable: true - will enable the creation of ClusterRole and ClusterRoleBinding for the service account mentioned in the snippet above\
rbacEnable: true and namespaceScope: true - will enable the creation of Role and RoleBinding which is more restrictive.

Configure logging

There are different logging modes and levels that can be adjusted.

values.yaml

lensesHq:
  logger:
    # Allowed values are: text | json
    mode: "text"

    # Allowed values are: info | debug
    level: "info"

Add chart repository

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

Installing HQ

Be aware that for the time being and for alpha purposes usage of --versionis mandatory when deploying Helm chart through Helm repository.

terminal

helm install lenses-hq lensesio/lenses-hq \
   --values values.yaml \
   --create-namespace --namespace lenses-hq \
   --version 6.0.8

Example Values files

Be aware that example of values.yaml shows only how all parameters should look at the end. Please fill them with correct values otherwise Helm installation might not be successful.

Example of values.yaml

More about default values for Lenses HQ Helm Chart can be found in values.yaml. An example is below:

values.yaml

resources:
  requests:
  #   cpu: 1
    memory: 4Gi
  limits:
  #   cpu: 2
    memory: 8Gi

image:
  repository: lensesio/lenses:latest

rbacEnable: true
namespaceScope: true

ingress:
  http:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
    host: example.com

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true. # optional
      cert:
        referenceFromSecret: true
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG
    administrators:
      - admin
      - [email protected]
      - [email protected]
    saml:
      enabled: true    # optional
      baseURL: ""
      entityID: ""
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: ""
        secretKeyName: ""
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          </md:EntityDescriptor>
  
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
      uiRootURL: "/"
      groupAttributeKey: "groups"
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    accessControlAllowCredentials: false
    secureSessionCookies: true
    tls:
      enabled: true    # optional
      cert:
      privateKey:
        secret:
          name: 
          key:
  # Find more details in https://docs.lenses.io/current/installation/kubernetes/helm/#helm-storage
  ## Postgres template example: "postgres://[username]:[pwd]@[host]:[port]/[database]?sslmode=require"
  storage:
    postgres:
      enabled: true
      host: [POSTGRES_HOST_URL]
      port: 5432
      username: [POSTGRES_USERNAME]
      database: [POSTGRES_USER_PWD]
      passwordSecret:
        type: "precreated"
        name: initContainer-2-db-secret
        key: password

What's next?

After the successful configuration and installation of HQ, the next steps would be:

Deploying and Agent
Configuring IAM roles / groups / policies

HQ

This page describe the Lenses Agent configuration.

HQ's configuration is defined in the config.yaml file

EULA

To accept the Lenses EULA, set the following in the lenses.conf file:

Without accepting the EULA the Agent will not start! See License.

It has the following top level groups:

Name

Required

Default

Type

Description

http

Yes

n/a

Configures everything involving the HTTP.

agents

Yes

n/a

Controls the agent handling.

database

Yes

n/a

Configures database settings.

logger

Yes

n/a

Sets the logger behaviour.

metrics

Yes

n/a

Controls the metrics settings.

license

Yes

n/a

Holds the license key.

auth

Yes

n/a

Configures authentication and authorisation

AuthConfig

Configures authentication and authorisation.

It has the following fields:

Name

Required

Default

Type

Description

administrators

[]

strings

Grants root access to principals.

saml

n/a

Contains SAML2 IdP configuration.

users

[]

Array

Creates initial users for password based authentication.

AuthConfig: administrators

Lists the names of the principals (users, service accounts) that have root access. Access control allows any API operation performed by such principals. Optional. If not set, it will default to [].

AuthConfig: saml

Contains SAML2 IdP configuration. Please refer here for its structure.

HTTPConfig

Configures everything involving the HTTP.

It has the following fields:

Name

Required

Default

Type

Description

address

Yes

n/a

string

Sets the address the HTTP server listens at.

accessControlAllowOrigin

["*"]

strings

Sets the value of the "Access-Control-Allow-Origin" header.

accessControlAllowCredentials

false

boolean

Sets the value of the "Access-Control-Allow-Credentials" header.

secureSessionCookies

true

boolean

Sets the "Secure" attribute on session cookies.

tls

Yes

n/a

Contains TLS configuration.

HTTPConfig: address

Sets the address the HTTP server listens at.

Example value: 127.0.0.1:80.

HTTPConfig: accessControlAllowOrigin

Sets the value of the "Access-Control-Allow-Origin" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to ["*"].

HTTPConfig: accessControlAllowCredentials

Sets the value of the "Access-Control-Allow-Credentials" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to false.

HTTPConfig: secureSessionCookies

Sets the "Secure" attribute on authentication session cookies. When set, a browser sends such cookies not over unsecured HTTP (expect for localhost). If running Lenses HQ over unsecured HTTP, set this to false. Optional. If not set, it will default to true.

HTTPConfig: tls

Contains TLS configuration. Please refer here for its structure.

SAMLConfig

Contains SAML2 IdP configuration.

It has the following fields:

Name

Required

Default

Type

Description

metadata

Yes

n/a

string

Contains the IdP issued XML metadata blob.

baseURL

Yes

n/a

string

Defines base URL of HQ for IdP redirects.

uiRootURL

/

string

Controls where to redirect to upon successful authentication.

entityID

Yes

n/a

string

Defines the Entity ID.

groupAttributeKey

groups

string

Sets the attribute name for group names.

userCreationMode

manual

string

Controls how the creation of users should be handled in relation to SSO information.

groupMembershipMode

manual

string

Controls how the management of a user's group membership should be handled in relation to SSO information.

SAMLConfig: metadata

Contains the IdP issued XML metadata blob.

Example value: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>.

SAMLConfig: baseURL

Defines the base URL of Lenses HQ; the IdP redirects back to here on success.

Example value: https://hq.example.com.

SAMLConfig: uiRootURL

Controls where the backend redirects to after having received a valid SAML2 assertion. Optional. If not set, it will default to /.

Example value: /.

SAMLConfig: entityID

Defines the Entity ID.

Example value: https://hq.example.com.

SAMLConfig: groupAttributeKey

Sets the attribute name from which group names are extracted in the SAML2 assertions. Different providers use different names. Okta, Keycloak and Google use "groups". OneLogin uses "roles". Azure uses "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups". Optional. If not set, it will default to groups.

Example value: groups.

SAMLConfig: userCreationMode

Controls how the creation of users should be handled in relation to SSO information. With the 'manual' mode, only users that currently exist in HQ can login. Users that do not exist are rejected. With the 'sso' mode, users that do not exist are automatically created. Allowed values are manual or sso. Optional. If not set, it will default to manual.

SAMLConfig: groupMembershipMode

Controls how the management of a user's group membership should be handled in relation to SSO information. With the 'manual' mode, the information about the group membership returned from an Identity Provider will not be used and a user will only be a member of groups that were explicitly assigned to him locally. With the 'sso' mode, group information from Identity Provider (IdP) will be used. On login, a user's group membership is set to the groups listed in the IdP. Groups that do not exist in HQ are ignored. Allowed values are manual or sso. Optional. If not set, it will default to manual.

AgentsConfig

Controls the agent handling.

It has the following fields:

Name

Required

Default

Type

Description

address

Yes

n/a

string

Sets the address the agent server listens at.

tls

Yes

n/a

Contains TLS configuration.

AgentsConfig: address

Sets the address the agent server listens at.

Example value: 127.0.0.1:3000.

AgentsConfig: tls

Contains TLS configuration. Please refer here for its structure.

TLSConfig

Contains TLS configuration.

It has the following fields:

Name

Required

Default

Type

Description

enabled

Yes

n/a

boolean

Enables or disables TLS.

cert

string

Sets the PEM formatted public certificate.

key

string

Sets the PEM formatted private key.

verboseLogs

false

boolean

Enables verbose TLS logging.

TLSConfig: enabled

Enables or disables TLS.

Example value: false.

TLSConfig: cert

Sets the PEM formatted public certificate. Optional. If not set, it will default to ``.

Example value: -----BEGIN CERTIFICATE----- EXampLeRanDoM ... -----END CERTIFICATE----- .

TLSConfig: key

Sets the PEM formatted private key. Optional. If not set, it will default to ``.

Example value: -----BEGIN PRIVATE KEY----- ExAmPlErAnDoM ... -----END PRIVATE KEY----- .

TLSConfig: verboseLogs

Enables additional logging of TLS settings and events at debug level. The information presented might be a bit too much for day to day use but can provide extra information for troubleshooting TLS configuration. Optional. If not set, it will default to false.

DatabaseConfig

Configures database settings.

It has the following fields:

Name

Required

Default

Type

Description

host

Yes

n/a

string

Sets the name of the host to connect to.

username

string

Sets the username to authenticate as.

password

string

Sets the password to authenticate as.

database

Yes

n/a

string

Sets the database to use.

schema

string

Sets the schema to use.

TLS

false

boolean

Enables TLS.

params

{}

DBConnectionParams

Provides fine-grained control.

DatabaseConfig: host

Sets the name of the host to connect to. A comma-separated list of host names is also accepted; each host name in the list is tried in order.

Example value: postgres:5432.

DatabaseConfig: username

Sets the username to authenticate as. Optional. If not set, it will default to ``.

Example value: johhnybingo.

DatabaseConfig: password

Sets the password to authenticate as. Optional. If not set, it will default to ``.

Example value: my-password.

DatabaseConfig: database

Sets the database to use.

Example value: my-database.

DatabaseConfig: schema

Sets the schema to use. Optional. If not set, it will default to ``.

Example value: my-schema.

DatabaseConfig: TLS

Enables TLS. In PostgreSQL connection string terms, setting TLS to false corresponds to sslmode=disable; setting TLS to true corresponds to sslmode=verify-full. For more fine-grained control, specify sslmode in the params which takes precedence. Optional. If not set, it will default to false.

Example value: true.

DatabaseConfig: params

Contains connection string parameters as key/values pairs. It allow fine-grained control of connection settings. The parameters can be found here: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS Optional. If not set, it will default to {}.

Example value: {"application_name":"example"}.

LoggerConfig

Sets the logger behaviour.

It has the following fields:

Name

Required

Default

Type

Description

mode

Yes

n/a

string

Controls the format of the logger's output.

level

info

string

Controls the level of the logger.

LoggerConfig: mode

Controls the format of the logger's output. Allowed values are text or json.

LoggerConfig: level

Controls the level of the logger. Allowed values are info or debug. Optional. If not set, it will default to info.

MetricsConfig

Controls the metrics settings.

It has the following fields:

Name

Required

Default

Type

Description

prometheusAddress

:9090

string

Sets the Prometheus address.

MetricsConfig: prometheusAddress

Sets the address at which Prometheus metrics are served. Optional. If not set, it will default to :9090.

License

Holds the license key.

It has the following fields:

Name

Required

Default

Type

Description

key

Yes

n/a

string

Sets the license key.

acceptEULA

Yes

fals

boolean

Accepts the

License: key

Sets the license key. An HQ key starts with "licensekey".

License: acceptEULA

Accepts the Lenses EULA.

How to convert Wizard Mode to Provisioning Mode

Migrating from Lenses Wizard Mode to Provision

Overview

Lenses 5.0+ introduces two primary methods for configuring your Lenses instance:

Wizard Mode: An interactive UI-based setup that appears when no Kafka brokers are configured
Provision Mode: A programmatic approach using provisioning.yaml configuration with(out) sidecar containers

This guide walks you through migrating from the Wizard Mode setup to a fully automated Provision configuration, enabling GitOps workflows and Infrastructure as Code practices.

Understanding the Migration Path

When to Consider Migration

You should migrate from Wizard Mode to Provision when you need:

Automated deployments and Infrastructure as Code
GitOps workflows for configuration management
Consistent environments across development, staging, and production
Version control for your Lenses configuration
Scalable deployment patterns for multiple Lenses Agent instances
Migrating from Lenses ⅘ to Lenses 6 Agent

Key Differences

Aspect

Wizard Mode

Provision Mode

Setup Method

Interactive UI

YAML configuration

Automation

Manual

Fully automated

Version Control

Not supported

Full Git integration

Secrets Management

Manual entry

Kubernetes secrets + file references

Deployment

One-time setup

Repeatable deployments

Configuration Updates

UI-based

Code-based with CI/CD

Pre-Migration Checklist

Before starting your migration, please check Tips: Before Upgrade

Assess Agent Connections that have to be migrated from Lenses 5

In the picture below it is visible that three connections has to be migrated to provisioning.yaml file:

Kafka
SchemaRegistry
KafkaConnect

in addition to that a new connection has to be made called LensesHQ.

Prepare Your Provision Configuration

Basic Structure

For Helm Deployment:

create a values.yaml file with the provision configuration enabled

For deployment from Archive create:

lenses-agent.conf and
provisioning.yaml file

Connections that provisioning is going to have:

Kafka;
SchemaRegistry;
KafkaConnect;
LensesHQ.

Configure Kafka Connection

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: lenses-agent-secret
        value: <your-HQ-generated-agentKey>
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: <your-HQ-address>
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false
      kafka:
        - name: kafka
          version: 1
          tags: [ "prod", "prod-1", "us"]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://<your-kafka-address>:9092
            metricsType:
             value: JMX
            metricsPort:
             value: 9999

For more Kafka connection details, such as using secure connections, please read Kafka

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: <your-HQ-generated-agentKey>
    port:
      value: 10000
    server:
      value: <your-HQ-address>
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://<your-kafka-address>:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1

Configure Additional Services Connections

Last two bits are:

Kafka Connect and
Schema Registry

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: lenses-agent-secret
        value: <your-HQ-generated-agentKey>
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: <your-HQ-address>
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false
      kafka:
        - name: kafka
          version: 1
          tags: [ "prod", "prod-1", "us"]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://<your-kafka-address>:9092
            metricsType:
             value: JMX
            metricsPort:
             value: 9999
      confluentSchemaRegistry:
        - name: schema-registry
          version: 1
          tags: [ "prod", "global" ]
          configuration:
            schemaRegistryUrls:
              value:
                - http://<your-schema-registry-address>:8081
      connect:
        - name: datalake-connect
          version: 1
          tags: [ "prod", "us" ]
          configuration:
            workers:
              value:
                - http://<your-kafka-connect-address>:8083

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: <your-HQ-generated-agentKey>
    port:
      value: 10000
    server:
      value: <your-HQ-address>
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://<your-kafka-address>:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1
confluentSchemaRegistry:
- configuration:
    schemaRegistryUrls:
      value:
      - http://<your-schemaregistry-address>:8081
  name: schema-registry
  tags:
  - prod
  - global
  version: 1
connect:
- configuration:
    workers:
      value:
      - http://<your-kafkaconnect-address>:8083
  name: datalake-connect
  tags:
  - prod
  - us
  version: 1

Prepare HQ Connection

Through last few steps, we covered configuring of:

Kafka
Schema Registry
Kafka Connect connections.

Last but not least and probably the most important one is creating HQ connection otherwise Agent won't be useable.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: lenses-agent-secret
        value: <your-HQ-generated-agentKey>
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: <your-HQ-address>
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: <your-HQ-generated-agentKey>
    port:
      value: 10000
    server:
      value: <your-HQ-address>
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://<your-kafka-address>:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1

License configuration is part of Lenses HQ from version 6.

Configure Database details

In case Postgres is being used:

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-1.postgres.svc.cluster.local
      port: 5432              # optional, defaults to 5432
      username: prod          
      password: external      # use "external" to manage it using secrets
      database: agent
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: local-postgres-pwd
          key: password
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: lenses-agent-secret
        value: agent_key_*
    ssl:
      enabled: false
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: lenses-hq.lenses.svc.cluster.local
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false
      kafka:
        - name: kafka
          version: 1
          tags: [ "prod", "prod-2", "eu"]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://testing-kafka-bootstrap.kafka.svc.cluster.local:9092
            metricsType:
             value: JMX
            metricsPort:
             value: 9999
      connect:
        - name: datalake-connect
          version: 1
          tags: [ "prod-2", "eu" ]
          configuration:
            workers:
              value:
                - http://testing-connect-connect.kafka-connect.svc.cluster.local:8083
      confluentSchemaRegistry:
        - name: schema-registry
          version: 1
          tags: [ "prod", "global" ]
          configuration:
            schemaRegistryUrls:
              value:
                - http://testing-schema-registry.schema-registry.svc.cluster.local:8081

You would have to use two files:

lenses-agent.conf
provisioning.yaml

lenses-agent.conf

# Auto-detected env vars

lenses.secret.file=/data/security.conf

# lenses.append.conf

lenses.storage.postgres.host="postgres-1.postgres.svc.cluster.local"
lenses.storage.postgres.database="agent"
lenses.storage.postgres.username="agent"
lenses.storage.postgres.password="pleasechangeme"
lenses.storage.postgres.port="5432"
lenses.provisioning.path="/mnt/provision-secrets"

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: <your-HQ-generated-agentKey>
    port:
      value: 10000
    server:
      value: <your-HQ-address>
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://<your-kafka-address>:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1
confluentSchemaRegistry:
- configuration:
    schemaRegistryUrls:
      value:
      - http://<your-schemaregistry-address>:8081
  name: schema-registry
  tags:
  - prod
  - global
  version: 1
connect:
- configuration:
    workers:
      value:
      - http://<your-kafkaconnect-address>:8083
  name: datalake-connect
  tags:
  - prod
  - us
  version: 1

H2 as a storage mechanism is available only from Agent v6.0.6

Be aware that H2 is not recommended for production environments

values.yaml

persistence:
  storageH2:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 5Gi

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: lenses-agent-secret
        value: agent_key_*
    ssl:
      enabled: false
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: lenses-hq.lenses.svc.cluster.local
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false
      kafka:
        - name: kafka
          version: 1
          tags: [ "prod", "prod-2", "eu"]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://testing-kafka-bootstrap.kafka.svc.cluster.local:9092
            metricsType:
             value: JMX
            metricsPort:
             value: 9999
      connect:
        - name: datalake-connect
          version: 1
          tags: [ "prod-2", "eu" ]
          configuration:
            workers:
              value:
                - http://testing-connect-connect.kafka-connect.svc.cluster.local:8083
      confluentSchemaRegistry:
        - name: schema-registry
          version: 1
          tags: [ "prod", "global" ]
          configuration:
            schemaRegistryUrls:
              value:
                - http://testing-schema-registry.schema-registry.svc.cluster.local:8081

You would have to use two files:

lenses-agent.conf
provisioning.yaml

lenses-agent.conf

# Auto-detected env vars
lenses.secret.file=/data/security.conf
# lenses.append.conf
lenses.provisioning.path="/mnt/provision-secrets"

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: <your-HQ-generated-agentKey>
    port:
      value: 10000
    server:
      value: <your-HQ-address>
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://<your-kafka-address>:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1
confluentSchemaRegistry:
- configuration:
    schemaRegistryUrls:
      value:
      - http://<your-schemaregistry-address>:8081
  name: schema-registry
  tags:
  - prod
  - global
  version: 1
connect:
- configuration:
    workers:
      value:
      - http://<your-kafkaconnect-address>:8083
  name: datalake-connect
  tags:
  - prod
  - us
  version: 1

Complete Migration Example

Here's a complete values.yaml and lenses-agent + provisioning.yaml example for a production migration:

For more Helm options, please check lenses-helm-chart repo.

values.yaml

rbacEnable: true
namespaceScope: true

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: lenses-agent-secret
        value: agent_key_*
    ssl:
      enabled: false
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: lenses-hq.lenses.svc.cluster.local
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false
      kafka:
        - name: kafka
          version: 1
          tags: [ "prod", "prod-2", "eu"]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://testing-kafka-bootstrap.kafka.svc.cluster.local:9092
            metricsType:
             value: JMX
            metricsPort:
             value: 9999
      connect:
        - name: datalake-connect
          version: 1
          tags: [ "prod-2", "eu" ]
          configuration:
            workers:
              value:
                - http://testing-connect-connect.kafka-connect.svc.cluster.local:8083
      confluentSchemaRegistry:
        - name: schema-registry
          version: 1
          tags: [ "prod", "global" ]
          configuration:
            schemaRegistryUrls:
              value:
                - http://testing-schema-registry.schema-registry.svc.cluster.local:8081
  storage:
    postgres:
      enabled: true
      host: postgres-1.postgres.svc.cluster.local
      port: 5432              # optional, defaults to 5432
      username: prod          
      password: external      # use "external" to manage it using secrets
      database: agent
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: local-postgres-pwd
          key: password

You would have to use two files:

lenses-agent.conf
provisioning.yaml

lenses-agent.conf

# Auto-detected env vars

lenses.kubernetes.pod.mem.request=128M
lenses.kubernetes.pod.mem.limit=1152M
lenses.jmx.port=9101
lenses.kubernetes.pod.liveness.initial.delay="60 seconds"
lenses.sql.execution.mode=KUBERNETES
lenses.topics.metrics=_kafka_lenses_metrics
lenses.provisioning.path="/mnt/provision-secrets"
lenses.topics.external.topology=__topology
lenses.kubernetes.pod.min.heap=128M
lenses.port=3030
lenses.kubernetes.pod.heap=1024M
lenses.topics.external.metrics=__topology__metrics

lenses.secret.file=/data/security.conf

# lenses.append.conf

lenses.storage.postgres.host="postgres-1.postgres.svc.cluster.local"
lenses.storage.postgres.database="agent"
lenses.storage.postgres.username="agent"
lenses.storage.postgres.password="pleasechangeme"
lenses.storage.postgres.port="5432"
lenses.provisioning.path="/mnt/provision-secrets"

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: <your-HQ-generated-agentKey>
    port:
      value: 10000
    server:
      value: <your-HQ-address>
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://<your-kafka-address>:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1
confluentSchemaRegistry:
- configuration:
    schemaRegistryUrls:
      value:
      - http://<your-schemaregistry-address>:8081
  name: schema-registry
  tags:
  - prod
  - global
  version: 1
connect:
- configuration:
    workers:
      value:
      - http://<your-kafkaconnect-address>:8083
  name: datalake-connect
  tags:
  - prod
  - us
  version: 1

Deployment and Testing

Deploy with Helm

Add the Lenses Helm repository:

helm repo add lenses-agent https://helm.repo.lenses.io
helm repo update

Deploy Lenses with your provision configuration:

helm install lenses-agent lenses/lenses-agent \
  --namespace lenses \
  --create-namespace \
  -f values.yaml

Monitor the deployment:

kubectl get pods -n lenses -w
kubectl logs -n lenses deployment/lenses-agent

Deploy through Archive

Download Archive

Link to archives can be found here: https://archive.lenses.io/lenses/6.0/agent/

Extract the archive using the following command

terminal

tar -xvf lenses-agent-latest-linux64.tar.gz -C lenses

Start Agent

terminal

bin/lenses lenses-agent.conf

Best Practices for Production

Security Considerations

Use Kubernetes secrets for sensitive data instead of inline values
Enable TLS for all Lenses HQ connections
Implement RBAC for Kubernetes and Lenses HQ & Agent access
Rotate credentials regularly

Operations Best Practices

Monitor resource usage of sidecar containers
Set resource limits to prevent resource monopolization
Implement health checks for the provision process
Use GitOps workflows for configuration management

Scaling Considerations

Plan for multiple environments (dev, staging, prod)
Implement configuration templates for reusability
Use Helm chart dependencies for complex deployments
Monitor deployment metrics and success rates

Conclusion

Migrating from Lenses Wizard Mode to Provision Mode enables Infrastructure as Code practices, better security management, and automated deployments. While the initial setup requires more configuration, the long-term benefits of automated, version-controlled, and repeatable deployments make this migration worthwhile for production environments.

The provision sidecar pattern ensures that your Lenses configuration is managed alongside your infrastructure code, enabling true GitOps workflows and reducing configuration drift between environments.

IAM Reference

This page describes the IAM Reference options.

Administration

service: administration

Resource Syntax

administration:connection:${Environment}/${ConnectionType}/${Connection}
administration:lenses-logs:${Environment}
administration:lenses-configuration:${Environment}
administration:setting:${Setting}

Operation

Resource Type

Description

CreateConnection

connection

ListConnections

connection

GetConnectionDetails

connection

UpdateConnection

connection

DeleteConnection

connection

GetLensesLogs

lenses-logs

GetLensesConfiguration

lenses-configuration

ListAgents

agent

GetAgentDetails

agent

UpdateAgent

agent

DeleteAgent

agent

GetSetting

setting

UpdateSetting

setting

Applications

service: applications

Resource Syntax

Operation

Resource Type

Description

RegisterApplication

external-application

UnregisterApplication

external-application

ListApplications

external-application

GetApplicationDetails

external-application

ListApplicationDependants

external-application

Alerts

service: alerts

Resource Syntax

alerts:alert:${Environment}/${AlertType}/${Alert}
alerts:rule:${Environment}/Infrastructure/KafkaBrokerDown
alerts:rule:${Environment}/DataProduced/red-app-going-slow

Operation

Resource Type

Description

CreateAlertRule

rule

DeleteAlertRule

rule

UpdateAlertRule

rule

ListAlertRules

rule

GetAlertRuleDetails

rule

ToggleAlertRule

rule

ListAlertEvents

alert-event

DeleteAlertEvents

alert-event

CreateChannel

alert-channel

ListChannels

alert-channel

GetChannelDetails

alert-channel

UpdateChannel

alert-channel

DeleteChannel

alert-channel

K2K

service: k2k

Resource Syntax

k2k:app:${Name}

Action

Resource Type

Description

CreateApp

app

DeleteApp

app

GetApp

app

ListApps

app

ManageOffsets

app

UpdateApp

app

UpsertApp

app

Audits

service: audit

Resource Syntax

audit:log:${Environment}
audit:channel:${Environment}/${AuditChannelType}/${AuditChannel}

Operation

Resource Type

Description

ListLogEvents

log

GetLogEventDetails

log

CreateChannel

channel

ListChannels

channel

GetChannelDetails

channel

UpdateChannel

channel

DeleteChannel

channel

ToggleChannel

channel

Data Policies

service: data-policies

Resource Syntax

data-policies:policy:${Environment}/${Policy}

Operation

Resource Type

Description

CreatePolicy

policy

ListPolicies

policy

GetPolicyDetails

policy

UpdatePolicy

policy

DeletePolicy

policy

ListPolicyDependants

policy

Environments

service: environments

Resource Syntax

environments:environment:${Environment}

Operation

Resource Type

Description

CreateEnvironment

environment

DeleteEnvironment

environment

ListEnvironments

environment

UpdateEnvironment

environment

AccessEnvironment

environment

GetEnvironmentDetails

environment

Permission which allows users to gain overview of more information about the environment such as metrics, versions and more.

Kafka Connections

service: environments

Resource Syntax

environments:kafka-connection:${Environment}/${Connection}

Operation

Resource Type

Description

GetKafkaConnectionDetails

environments

ListKafkaConnections

environments

UpsertKafkaConnection

environments

Create or update a Kafka Connection

DeleteKafkaConnection

environments

Governance

service: governance

Resource Syntax

governance:request:${Environment}/${ActionType}/*
governance:rule:${Environment}/${RuleCategory}/*

Operation

Resource Type

Description

CreateRequest

request

ListRequests

request

GetRequestDetails

request

ApproveRequest

request

DenyRequest

request

GetRuleDetails

rule

UpdateRule

rule

IAM

service: iam

Resource Syntax

iam:role:${Role}
iam:group:${Group}
iam:user:${Username}
iam:service-account:${ServiceAccount}

Operation

Resource Type

Description

CreateRole

role

DeleteRole

role

UpdateRole

role

ListRoles

role

ListRoleDependants

role

GetRoleDetails

role

CreateGroup

group

DeleteGroup

group

UpdateGroup

group

ListGroups

group

ListGroupDependants

group

GetGroupDetails

group

CreateUser

user

DeleteUser

user

UpdateUser

user

ListUsers

user

ListUserDependants

user

GetUserDetails

user

CreateServiceAccount

service account

DeleteServiceAccount

service account

UpdateServiceAccount

service account

ListServiceAccounts

service account

ListServiceAccountDependants

service account

GetServiceAccountDetails

service account

Kafka Connect

service: kafka-connect

Resource Syntax

kafka-connect:connector:${Environment}/${KafkaConnectCluster}/${Connector}
kafka-connect:cluster:${Environment}/${KafkaConnectCluster}

Example role permission

name: global-connector-operator
policy:
  - action:
      - iam:List*
      - iam:Get*
    resource: iam:*
    effect: allow
  - action:
      - environments:Get*
      - environments:List*
      - environments:AccessEnvironment
    resource: environments:*
    effect: allow
  - action:
      - kafka-connect:List*
      - kafka-connect:GetClusterDetails
      - kafka-connect:GetConnectorDetails
      - kafka-connect:StartConnector
      - kafka-connect:StopConnector
    resource:
      - kafka-connect:cluster:*/*
      - kafka-connect:connector:*/*/*
    effect: allow

Operation

Resource Type

Description

CreateConnector

connector

ListConnectors

connector

ListConnectors

connector

GetConnectorConfiguration

connector

UpdateConnectorConfiguration

connector

DeleteConnector

connector

StartConnector

connector

StopConnector

connector

ListConnectorDependants

connector

ListClusters

cluster

GetClusterDetails

cluster

DeployConnectors

cluster

Kafka

service: kafka

Resource Syntax

kafka:topic:${Environment}/${KafkaCluster}/${Topic}
kafka:acl:${Environment}/${KafkaCluster}/${AclResourceType}/* or kafka:acl:${Environment}/${KafkaCluster}/${AclResourceType}/${PrincipalType}/${Principal}
kafka:quota:${Environment}/${KafkaCluster}/${QuotaType}/* or
kafka:quota:${Environment}/${KafkaCluster}/clients
kafka:quota:${Environment}/${KafkaCluster}/users-default
kafka:quota:${Environment}/${KafkaCluster}/client/${ClientID}
kafka:quota:${Environment}/${KafkaCluster}/user/${Username}
kafka:quota:${Environment}/${KafkaCluster}/user/${Username}/client/${ClientID}
kafka:quota:${Environment}/${KafkaCluster}/user-client/${Username}/${ClientID}
kafka:quota:${Environment}/${KafkaCluster}/user/${Username}/client/*
kafka:quota:${Environment}/${KafkaCluster}/user-all-clients/${Username}

Example role permission

name: example
policy:
  - action:
      - kafka:ListTopics
      - kafka:GetTopicDetails 
    resource: 
      - kafka:topic:my_env/kafka/my_topic

Operation

Resource Type

Description

CreateTopic

topic

DeleteTopic

topic

ListTopics

topic

GetTopicDetails

topic

UpdateTopicDetails

topic

ReadTopicData

topic

WriteTopicData

topic

DeleteTopicData

topic

ListTopicDependants

topic

List visibility of all entities that depend on this entity e.g. ListTopicDependants means that you'll be able to see (i.e. List) all consumer groups that read from that topic regardless of what your specific consumer group permissions.

CreateAcl

acl

GetAclDetails

acl

UpdateAcl

acl

DeleteAcl

acl

CreateQuota

quota

ListQuotas

quota

GetQuotaDetails

quota

UpdateQuota

quota

DeleteQuota

quota

DeleteConsumerGroup

consumer-group

UpdateConsumerGroup

consumer-group

ListConsumerGroups

consumer-group

GetConsumerGroupDetails

consumer-group

ListConsumerGroupDependants

consumer-group

Kubernetes

service: kubernetes

Resource Syntax

kubernetes:cluster:${Environment}/${KubernetesCluster}
kubernetes:namespace:${Environment}/${KubernetesCluster}/${KubernetesNamespace}

Operation

Resource Type

Description

Example

ListClusters

cluster

GetClusterDetails

cluster

ListNamespaces

namespace

DeployApps

namespace

Registry

service: registry

Resource Syntax

schemas:registry:${Environment}/${SchemaRegistry}

Operation

Resource Type

Description

GetRegistryConfiguration

registry

UpdateRegistryConfiguration

registry

Schemas

service: schemas

Resource Syntax

schemas:schema:${Environment}/${SchemaRegistry}/${Schema}

Operation

Resource Type

Description

CreateSchema

schema

DeleteSchema

schema

UpdateSchema

schema

GetSchemaDetails

schema

ListSchemas

schema

ListSchemaDependants

schema

SQL Streaming

service: sql-streaming

Resource Syntax

sql-streaming:sql-processor:${Environment}/${KubernetesCluster}/${KubernetesNamespace}/${SqlProcessor}
For IN_PROC processors sql-streaming:sql-processor:${Environment}/lenses-in-process/default/${SqlProcessor}

Available Actions

Resource Type

Description

CreateProcessor

sql-processor

ListProcessors

sql-processor

GetProcessorDetails

sql-processor

GetProcessorSql

sql-processor

UpdateProcessorSql

sql-processor

DeleteProcessor

sql-processor

StartProcessor

sql-processor

StopProcessor

sql-processor

ScaleProcessor

sql-processor

GetProcessorLogs

sql-processor

ListProcessorDependants

sql-processor

Configuration Reference

This page lists the available configurations in Lenses Agent.

Set in lenses.conf

Basics

Reference documentation of all configuration and authentication options:

Key

Description

Default

Type

Required

lenses.eula.accept

Accept the

false

boolean

yes

lenses.ip

Bind HTTP at the given endpoint. Use in conjunction with lenses.port

0.0.0.0

string

lenses.port

The HTTP port to listen for API, UI and WS calls

9991

int

lenses.jmx.port

Bind JMX port to enable monitoring Lenses

int

lenses.root.path

The path from which all the Lenses URLs are served

string

lenses.secret.file

The full path to security.conf for security credentials

security.conf

string

lenses.sql.execution.mode

Streaming SQL mode IN_PROC (test mode) or KUBERNETES (prod mode)

IN_PROC

string

lenses.offset.workers

Number of workers to monitor topic offsets

int

lenses.kafka.control.topics

An array of topics to be treated as “system topics”

list

array

lenses.grafana

Add your Grafana url i.e. http://grafanahost:port

string

lenses.api.response.cache.enable

If enabled, it disables client cache on the Lenses API HTTP responses by adding these HTTP Headers: Cache-Control: no-cache, no-store, must-revalidate, Pragma: no-cache, and Expires: -1.

false

boolean

lenses.workspace

Directory to write temp files. If write access is denied, Lenses will fallback to /tmp.

/run

string

lenses.connections.webhook.whitelist

This configuration key allows you to specify a whitelist of allowed IP ranges and hostnames for webhook connections. Only addresses matching the whitelist will be permitted for webhook connections.

The value should be a list of strings, where each string can be:

An IPv4 address (e.g., "192.168.1.10")
An IPv4 CIDR range (e.g., "192.168.1.0/24")
An IPv6 address (e.g., "2001:db8::1")
An IPv6 CIDR range (e.g., "2001:db8::/32")
A hostname pattern (e.g., "*.trusted.com", "localhost", "api.example.com"

array

Default system topics

System or control topics are created by services for their internal use. Below is the list of built-in configurations to identify them.

_schemas
__consumer_offsets
_kafka_lenses_
lsql_*
lsql-*
__transaction_state
__topology
__topology__metrics
_confluent*
*-KSTREAM-*
*-TableSource-*
*-changelog
__amazon_msk*

Wildcard (*) is used to match any name in the path to capture a list of topics not just one. When the wildcard is not specified, Lenses matches on the entry name provided.

Security

TLS

Key

Description

Default

lenses.access.control.allow.methods

HTTP verbs allowed in cross-origin HTTP requests

GET,POST,PUT,DELETE,OPTIONS

lenses.access.control.allow.origin

Allowed hosts for cross-origin HTTP requests

*

lenses.allow.weak.ssl

Allow https:// with self-signed certificates

false

lenses.ssl.keystore.location

The full path to the keystore file used to enable TLS on Lenses port

lenses.ssl.keystore.password

Password for the keystore file

lenses.ssl.key.password

Password for the ssl certificate used

lenses.ssl.enabled.protocols

Version of TLS protocol to use

TLSv1.2

lenses.ssl.algorithm

X509 or PKIX algorithm to use for TLS termination

SunX509

lenses.ssl.cipher.suites

Comma separated list of ciphers allowed for TLS negotiation

Kerberos

Key

Description

Default

lenses.security.kerberos.service.principal

The Kerberos principal for Lenses to use in the SPNEGO form: HTTP/[email protected]

lenses.security.kerberos.keytab

Path to Kerberos keytab with the service principal. It should not be password protected

lenses.security.kerberos.debug

Enable Java’s JAAS debugging information

false

Persistent storage

Common

Key

Description

Default

Type

Required

lenses.storage.hikaricp.[*]

To pass additional properties to HikariCP connection pool

Postgres

Key

Description

Default

Type

Required

lenses.storage.postgres.host

Host of PostgreSQL server for Lenses to use for persistence

string

lenses.storage.postgres.port

Port of PostgreSQL server for Lenses to use for persistence

5432

integer

lenses.storage.postgres.username

Username for PostgreSQL database user

string

lenses.storage.postgres.password

Password for PostgreSQL database user

string

lenses.storage.postgres.database

PostgreSQL database name for Lenses to use for persistence

string

lenses.storage.postgres.schema

PostgreSQL schema name for Lenses to use for persistence

"public"

string

lenses.storage.postgres.properties.[*]

To pass additional properties to PostgreSQL JDBC driver

Microsoft SQL Server

Set in security.conf

Key

Description

Default

Type

Required

lenses.storage.msssql.host

Specifies the hostname or IP address of the Microsoft SQL Server instance

string

yes

lenses.storage.mssql.port

Specifies the TCP port number that the Lenses application uses to connect to a Microsoft SQL Server database

int

yes

lenses.storage.mssql.schema

Specifies the database schema Lenses uses within Microsoft SQL Server

string

yes

lenses.storage.mssql.database

Specifies the Microsoft SQL server database Lenses connects to

string

yes

lenses.storage.mssql.username

Specifies the username that the Lenses application uses to authenticate with the Microsoft SQL Server database

string

yes

lenses.storage.mssql.password

Specifies the password that the Lenses application uses to authenticate with the Microsoft SQL Server database

string

yes

lenses.storage.mssql.properties

Allows additional properties to be set for the Microsoft SQL Servicer JDBC drive

CommentShare feedback on the editor

Schema registries

If the records schema is centralized, the connectivity to Schema Registry nodes is defined by a Lenses Connection.

There are two static config entries to enable/disable the deletion of schemas:

Key

Description

Type

lenses.schema.registry.delete

Allow schemas to be deleted. Default is false

boolean

lenses.schema.registry.cascade.delete

Deletes associated schemas when a topic is deleted. Default is false

boolean

Deployments

Options for specific deployment targets:

Global options
Kubernetes

Global options

Common settings, independently of the underlying deployment target:

Key

Description

Default

lenses.deployments.events.buffer.size

Buffer size for events coming from Deployment targets such as Kubernetes

10000

lenses.deployments.errors.buffer.size

Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes

1000

Kubernetes

Kubernetes connectivity is optional. Minimum supported K8 version 0.11.10. All settings are string.

Key

Description

Default

lenses.kubernetes.processor.image.name

The url for the streaming SQL Docker for K8

lensesioextra/sql-processor

lenses.kubernetes.processor.image.tag

The version/tag of the above container

5.2

lenses.kubernetes.config.file

The path for the kubectrl config file

/home/lenses/.kube/config

lenses.kubernetes.pull.policy

Pull policy for K8 containers: IfNotPresent or Always

IfNotPresent

lenses.kubernetes.service.account

The service account for deployments. Will also pull the image

default

lenses.kubernetes.init.container.image.name

The docker/container repository url and name of the Init Container image used to deploy applications to Kubernetes

lensesio/lenses-cli

lenses.kubernetes.init.container.image.tag

The tag of the Init Container image used to deploy applications to Kubernetes

5.2.0

lenses.kubernetes.watch.reconnect.limit

How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable

10

lenses.kubernetes.watch.reconnect.interval

How often to wait between Kubernetes Watcher reconnection attempts expressed in milliseconds

5000

lenses.kubernetes.websocket.timeout

How long to wait for a Kubernetes Websocket response expressed in milliseconds

15000

lenses.kubernetes.websocket.ping.interval

How often to ping Kubernetes Websocket to check it’s alive expressed in milliseconds

30000

lenses.kubernetes.pod.heap

The max amount of memory the underlying Java process will use

900M

lenses.kubernetes.pod.min.heap

The initial amount of memory the underlying Java process will allocate

128M

lenses.kubernetes.pod.mem.request

The value will control how much memory resource the Pod Container will request

128M

lenses.kubernetes.pod.mem.limit

The value will control the Pod Container memory limit

1152M

lenses.kubernetes.pod.cpu.request

The value will control how much cpu resource the Pod Container will request

null

lenses.kubernetes.pod.cpu.limit

The value will control the Pod Container cpu limit

null

lenses.kubernetes.namespaces

Object setting a list of Kubernetes namespaces that Lenses will see for each of the specified and configured cluster

null

lenses.kubernetes.pod.liveness.initial.delay

Amount of time Kubernetes will wait to check Processor’s health for the first time. It can be expressed like 30 second, 2 minute or 3 hour, mind the time unit is singular

60 second

lenses.deployments.events.buffer.size

Buffer size for events coming from Deployment targets such as Kubernetes

10000

lenses.deployments.errors.buffer.size

Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes

1000

lenses.kubernetes.config.reload.interval

Time interval to reload the Kubernetes configuration file. Expressed in milliseconds.

30000

SQL snapshot (Explore & Studio)

Optimization settings for SQL queries.

Key

Description

Type

Default

lenses.sql.settings.max.size

Restricts the max bytes that a kafka sql query will return

long

20971520 (20MB)

lenses.sql.settings.max.query.time

Max time (in msec) that a sql query will run

int

3600000 (1h)

lenses.sql.settings.max.idle.time

Max time (in msec) for a query when it reaches the end of the topic

int

5000 (5 sec)

lenses.sql.settings.show.bad.records

By default show bad records when querying a kafka topic

boolean

true

lenses.sql.settings.format.timestamp

By default convert AVRO date to human readable format

boolean

true

lenses.sql.settings.live.aggs

By default allow aggregation queries on kafka data

boolean

true

lenses.sql.sample.default

Number of messages to sample when live tailing a kafka topic

int

2/window

lenses.sql.sample.window

How frequently to sample messages when tailing a kafka topic

int

200 msec

lenses.sql.websocket.buffer

Buffer size for messages in a SQL query

int

10000

lenses.metrics.workers

Number of workers for parallelising SQL queries

int

16

lenses.kafka.ws.buffer.size

Buffer size for WebSocket consumer

int

10000

lenses.kafka.ws.max.poll.records

Max number of kafka messages to return in a single poll()

long

1000

lenses.sql.state.dir

Folder to store KStreams state.

string

logs/sql-kstream-state

lenses.sql.udf.packages

The list of allowed java packages for UDFs/UDAFs

array of strings

["io.lenses.sql.udf"]

Lenses internal Kafka topics

Lenses requires these Kafka topics to be available, otherwise, it will try to create them. The topics can be created manually before Lenses is run, or allow Lenses the correct Kafka ACLs to create the topics:

Key

Description

Partition

Replication

Default

Compacted

Retention

lenses.topics.external.topology

Topic for applications to publish their topology

1

3 (recommended)

__topology

yes

N/A

lenses.topics.external.metrics

Topic for external application to publish their metrics

1

3 (recommended)

__topology__metrics

1 day

lenses.topics.metrics

Topic for SQL Processor to send the metrics

1

3 (recommended)

_kafka_lenses_metrics

To allow for fine-grained control over the replication factor of the three topics, the following settings are available:

Key

Description

Default

lenses.topics.replication.external.topology

Replication factor for the lenses.topics.external.topology topic

lenses.topics.replication.external.metrics

Replication factor for the lenses.topics.external.metrics topic

lenses.topics.replication.metrics

Replication factor for the lenses.topics.metrics topic

When configuring the replication factor for your deployment, it's essential to consider the requirements imposed by your cloud provider. Many cloud providers enforce a minimum replication factor to ensure data durability and high availability. For example, IBM Cloud mandates a minimum replication factor of 3. Therefore, it's crucial to set the replication factor for the Lenses internal topics to at least 3 when deploying Lenses on IBM Cloud.

Advanced

All time configuration options are in milliseconds.

Key

Description

Type

Default

lenses.interval.summary

How often to refresh kafka topic list and configs

long

10000

lenses.interval.consumers.refresh.ms

How often to refresh kafka consumer group info

long

10000

lenses.interval.consumers.timeout.ms

How long to wait for kafka consumer group info to be retrieved

long

300000

lenses.interval.partitions.messages

How often to refresh kafka partition info

long

10000

lenses.interval.type.detection

How often to check kafka topic payload info

long

30000

lenses.interval.user.session.ms

How long a client-session stays alive if inactive (4 hours)

long

14400000

lenses.interval.user.session.refresh

How often to check for idle client sessions

long

60000

lenses.interval.topology.topics.metrics

How often to refresh topology info

long

30000

lenses.interval.schema.registry.healthcheck

How often to check the schema registries health

long

30000

lenses.interval.schema.registry.refresh.ms

How often to refresh schema registry data

long

30000

lenses.interval.metrics.refresh.zk

How often to refresh ZK metrics

long

5000

lenses.interval.metrics.refresh.sr

How often to refresh Schema Registry metrics

long

5000

lenses.interval.metrics.refresh.broker

How often to refresh Kafka Broker metrics

long

5000

lenses.interval.metrics.refresh.connect

How often to refresh Kafka Connect metrics

long

30000

lenses.interval.metrics.refresh.brokers.in.zk

How often to refresh from ZK the Kafka broker list

long

5000

lenses.interval.topology.timeout.ms

Time period when a metric is considered stale

long

120000

lenses.interval.audit.data.cleanup

How often to clean up dataset view entries from the audit log

long

300000

lenses.audit.to.log.file

Path to a file to write audits to in JSON format.

string

lenses.interval.jmxcache.refresh.ms

How often to refresh the JMX cache used in the Explore page

long

180000

lenses.interval.jmxcache.graceperiod.ms

How long to pause for when a JMX connectity error occurs

long

300000

lenses.interval.jmxcache.timeout.ms

How long to wait for a JMX response

long

500

lenses.interval.sql.udf

How often to look for new UDF/UDAF (user defined [aggregate] functions)

long

10000

lenses.kafka.consumers.batch.size

How many consumer groups to retrieve in a single request

Int

500

lenses.kafka.ws.heartbeat.ms

How often to send heartbeat messages in TCP connection

long

30000

lenses.kafka.ws.poll.ms

Max time for kafka consumer data polling on WS APIs

long

10000

lenses.kubernetes.config.reload.interval

Time interval to reload the Kubernetes configuration file.

long

30000

lenses.kubernetes.watch.reconnect.limit

How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable

long

10

lenses.kubernetes.watch.reconnect.interval

How often to wait between Kubernetes Watcher reconnection attempts

long

5000

lenses.kubernetes.websocket.timeout

How long to wait for a Kubernetes Websocket response

long

15000

lenses.kubernetes.websocket.ping.interval

How often to ping Kubernetes Websocket to check it’s alive

long

30000

lenses.akka.request.timeout.ms

Max time for a response in an Akka Actor

long

10000

lenses.sql.monitor.frequency

How often to emit healthcheck and performance metrics on Streaming SQL

long

10000

lenses.audit.data.access

Record dataset access as audit log entries

boolean

true

lenses.audit.data.max.records

How many dataset view entries to retain in the audit log. Set to -1 to retain indefinitely

int

500000

lenses.explore.lucene.max.clause.count

Override Lucene’s maximum number of clauses permitted per BooleanQuery

int

1024

lenses.explore.queue.size

Optional setting to bound Lenses internal queue used by the catalog subsystem. It needs to be positive integer or it will be ignored.

int

N/A

lenses.interval.kafka.connect.http.timeout.ms

How long to wait for Kafka Connect response to be retrieved

int

10000

lenses.interval.kafka.connect.healthcheck

How often to check the Kafka health

int

15000

lenses.interval.schema.registry.http.timeout.ms

How long to wait for Schema Registry response to be retrieved

int

10000

lenses.interval.zookeeper.healthcheck

How often to check the Zookeeper health

int

15000

lenses.ui.topics.row.limit

The number of Kafka records to load automatically when exploring a topic

int

200

lenses.deployments.connect.failure.alert.check.interval

Time interval in seconds to check the connector failure grace period has completed. Used by the Connect auto-restart failed connectors functionality. It needs too be a value between (1,600].

int

10

lenses.provisioning.path

Folder on the filesystem containing the provisioning data. See [provisioning docs](link to provisioning docs) for further details

string

lenses.provisioning.interval

Time interval in seconds to check for changes on the provisioning resources

int

lenses.schema.registry.client.http.retryOnTooManyRequest

When enabled, Lenses will retry a request whenever the schema registry returns a 429 Too Many Requests

boolean

lenses.schema.registry.client.http.maxRetryAwait

Max amount of time to wait whenever a 429 Too Many Requests is returned.

duration

lenses.schema.registry.client.http.maxRetryCount

Max retry count whenever a 429 Too Many Requests is returned.

integer

lenses.schema.registry.client.http.rate.type

Specifies if http requests to the configured schema registry should be rate limited. Can be "session" or "unlimited"

"unlimited" | "session"

lenses.schema.registry.client.http.rate.maxRequests

Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.

integer

N/A

lenses.schema.registry.client.http.rate.window

Whenever the rate limiter is "session" this configuration will determine the duration of the window used.

duration

N/A

lenses.schema.connect.client.http.retryOnTooManyRequest

Retry a request whenever a connect cluster returns a 429 Too Many Requests

boolean

lenses.schema.connect.client.http.maxRetryAwait

Max amount of time to wait whenever a 429 Too Many Requests is returned.

duration

lenses.schema.connect.client.http.maxRetryCount

Max retry count whenever a 429 Too Many Requests is returned.

integer

lenses.connect.client.http.rate.type

Specifies if http requests to the configured connect cluster should be rate limited. Can be "session" or "unlimited"

"unlimited" | "session"

lenses.connect.client.http.rate.maxRequests

Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.

integer

N/A

lenses.connect.client.http.rate.window

Whenever the rate limiter is "session" this configuration will determine the duration of the window used.

duration

N/A

Connectors topology

Control how Lenses identifies your connectors in the Topology view. Catalogue your connector types, set their icons, and control how Lenses extracts the topics used by your connectors.

Lenses comes preconfigured for some of the popular connectors as well as the Stream Reactor connectors. If you see that Lenses doesn’t automatically identify your connector type then use the lenses.connectors.info setting to register it with Lenses.

Add a new HOCON object {} for every new Connector in your lenses.connectors.info list :

  lenses.connectors.info = [
      {
        class.name = "The connector full classpath"
        name = "The name which will be presented in the UI"
        instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
        sink = true
        extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
        icon = "file.png"
        description = "A description for the connector"
        author = "The connector author"
      }
  ]

This configuration allows the connector to work with the topology graph, and also have the RBAC rules applied to it.

Source example

To extract the topic information from the connector configuration, source connectors require an extra configuration. The extractor class should be: io.lenses.config.kafka.connect.SimpleTopicsExtractor. Using this extractor requires an extra property configuration. It specifies the field in the connector configuration which determines the topics data is sent to.

Here is an example for the file source:

  lenses.connectors.info = [
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSource"
      name = "File"
      instance = "file"
      sink = false
      property = "topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
    }
  ]

Sink example

An example of a Splunk sink connector and a Debezium SQL server connector

  lenses.connectors.info = [
    {
      class.name = "com.splunk.kafka.connect.SplunkSinkConnector"
      name = "Splunk Sink",
      instance = "splunk.hec.uri"
      sink = true,
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "splunk.png",
      description = "Stores Kafka data in Splunk"
      docs = "https://github.com/splunk/kafka-connect-splunk",
      author = "Splunk"
    },
    {
      class.name = "io.debezium.connector.sqlserver.SqlServerConnector"
      name = "CDC MySQL"
      instance = "database.hostname"
      sink = false,
      property = "database.history.kafka.topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "debezium.png"
      description = "CDC data from RDBMS into Kafka"
      docs = "//debezium.io/docs/connectors/mysql/",
      author = "Debezium"
    }
  ]

External Applications

Key

Description

Default

Type

Required

apps.external.http.state.refresh.ms

When registering a runner for external app, a health-check interval can be specified. If it is not, this default interval is used (value in milliseconds)

30000

int

apps.external.http.state.cache.expiration.ms

Last known state of the runner is stored in a cache. The entries in the cache are being invalidated after a time that is defined by following configuration key (value in milliseconds). This value should not be lower than the apps.external.http.state.refresh.ms value.

60000

int