What’s New?

For versions 4.0 to 5.4 see our legacy documentation.

5.5.10

Improvements

• Audit Logs include the list of groups presented by IdP for the user

• The group listing API endpoint now includes a list of users

5.5.9

Improvements

• Optimise kubernetes event handling

• Add extra logging for queue processing and event handling

5.5.8

Improvements

• Optimise topic auto-detection audit logging to avoid duplicate entries

• Optimise logging (adjust UDAF for intellisense polluting the logs, better actor mailbox logging)

• Improvements to the connector verification logic when Lenses has to mock topics or topics.regex

5.5.7

Improvements

This version improves the fetching of schemas from Schema Registries. The related subsystem has been re-worked to provide better error handling, fewer requests to the Schema Registry, and support rate-limiting. Find out how to configure rate limiting.

5.5.6

Improvements

• The S3 backup/restore functionality now supports the latest version of the Stream Reactor S3 connector plugin.

• New users coming from LDAP will not be created unless they have groups coming from LDAP matching Lenses groups. Users can still be created manually by an administrator.

5.5.5

Improvements

• Improve performance of the data catalogue. Lenses should now be many times faster to detect topics and their serialization, and use less memory and CPU time. For teams with Kafka clusters that have thousands of schemas, the startup time will also improve. For teams with tens of thousands of schemas, consumers, and partitions, software stability will also improve.

• Bring back the restart task button for paused connectors. This undocumented behaviour of Kafka Connect allows users to stop a connector’s consumer group, so they can reset offsets. For Kafka Connect 3.5 or later the new STOP connector API and corresponding button in Lenses can have the same effect.

• Compress schemas before sending them to the Schema Registry. This allows to send larger schemas to the Schema Registry as the limit is on the size of the request rather than the schema itself.

• Improvements to the Skip Validation option for inserting JSON messages, to allow for less strict (but still valid) schemas for inserted messages.

5.5.4

New Features

• Add STOP operation (button) for Connectors. The STOP operation requires Kafka Connect 3.5 or greater

• Allow to skip schema validation when inserting to JSON topics

Improvements

• Connector search is now case-insensitive

• Allow to type to search groups when creating service accounts

• Show masked passwords when editing a connector (regression in 5.5.3)

Fixes

• Filtering connectors by type doesn’t work

• When there were at least two Connect clusters with at least one connector with common name in both clusters, filtering connectors returns incorrect or multiple results

• Validating connectors with passwords may not work (regression in 5.5.3)

5.5.3

New Features

Support for case-insensitive LDAP users

Whilst Lenses users are case-sensitive, LDAP most of the time performs case-insensitive searches on user accounts. This can lead to users who try to login to Lenses with different casing in their username (e.g., user and USER) to get duplicate accounts.

We added the option lenses.security.ldap.case.sensitive with a default value of true. It can be switched to false in which case Lenses will treat usernames from LDAP as case-insensitive and always converting to lowercase.

Improvements

• Upgrade the AWS IAM library to better support service account roles inside EKS

• Upgrade libraries with known CVEs —not affecting Lenses in either way

Fixes

• Fix Grafana link not showing up on sidebar

• Fix a case where some sensitive data might leak in the logs

• Fix filtering by connector name causing the connector screen to crash if a connect cluster is offline

5.5.2

Improvements

• The connectors’ screen will not mask passwords if they are referencing a secret from a secret provider.

Fixes

• Fix regression where connectors’ passwords were not masked.

5.5.1

Improvements

• Authentication:

  • Enhanced authentication to reject with a 401 status code when the user lacks any attached groups in the IdP (Identity Provider).

  • Improved authentication flow, allowing an authenticated SSO (Single Sign-On) user to log in even if there isn’t a corresponding group in Lenses.

• Documentation Enhancement:

  • New SQL processor page with direct links to the latest documentation and support resources for user convenience.

Fixes

• Deployment Issue:

  • Addressed a bug introduced in Lenses git ops deployment* version 5.5.0, resolving provisioning issues experienced in certain deployment scenarios.

• SSO Authentication Fix:

  • Corrected SSO authentication behavior. When an SSO user is configured to overwrite the IdP groups, Lenses now correctly refrains from extracting groups from the IdP.

5.5.0

New Features

Kafka Connectors as Code

Lenses now introduces support for managing Kafka connectors as code. With this feature, you can define your connectors in a YAML file and seamlessly deploy them to Lenses. This capability is accessible via both the Lenses CLI and the Lenses UI. This release marks the commencement of our journey towards a more declarative and automated approach to managing Kafka and Lenses resources.

Consumer Group Management

In this version, Lenses introduces support for deleting consumer group offsets and entire consumer groups, enhancing flexibility and control over consumer group management.

Generic SSO Provider

Lenses provides support for a few SSO providers out of the box like Google, Okta, etc. In this release, Lenses introduces a generic SSO provider, enabling users to integrate with any SSO provider that supports the SAML 2.0 protocol. This feature is configurable via the lenses.conf file under lenses.security.saml.idp.provider.

Enhancements

Kafka Message Replay

The Kafka message replay feature receives an enhancement, now enabling users to replay messages from a specific offset. This functionality is accessible from both the Lenses topic screen and the Lenses SQL studio screen, providing greater precision in message replay operations.

Consumer Group Offsets Data Link

Users can now seamlessly navigate from the consumer group offsets screen to the data of the topic that the consumer group offset points to, enhancing visibility and ease of data exploration.

Audits to log file

Lenses now provides the capability to log audit events to its log file, enabling users to store audit logs locally for compliance and security purposes. This feature is configurable via the lenses.conf file under lenses.audit.to.log.file.

Lenses Internal Topics Replication Factor

To ensure compatibility with cloud providers such as IBM, where a minimum replication factor is mandated, Lenses now allows the configuration of the replication factor for its internal topics. This setting can be configured in the lenses.conf file under lenses.internal.topics.replication.***.

Bug Fixes

External Applications via Lenses SDK

The Lenses SDK, a thin client facilitating the monitoring and tracking of external applications connected to Kafka within Lenses topology, has been enhanced in this release. An issue where the application’s status in Lenses was not updated correctly has been resolved.

S3 Backup-Restore for JSON Payloads

In this release, a bug affecting the S3 backup-restore feature for JSON payloads has been rectified. Previously, the feature encountered issues due to the Connect converter enforcing schema on JSON payloads, leading to incorrect functionality. This bug has been addressed to ensure seamless backup and restoration of JSON data via S3.

Ugrade Notes

Lenses 5.5 is an incremental release which brings in new features and improvements.

Upgrading from 5.0 or later does not require any static configuration change but if you have automated the creation of any AWS connection, then you will have to adjust the provisioning section of your Helm chart, or your CICD, or —if you use the API directly— your API calls.

If you are upgrading from version 4.3 or older, you need to follow the upgrade procedure for Lenses 5.0 as well as the rest of the instructions that follow.

Breaking Changes and Caution Items

Lenses upgrades (except patch releases) are not backwards compatible. It is best practice to take a backup of the Lenses database before an upgrade.

New provisioning API [caution]

With Lenses 5.3 the provisioning API was introduced. This new API can be used to create or update the connections landscape. The old provisioning methods could only create the connection landscape (first run).

What this means, is that now the Helm chart or a CICD process can be used to manage Lenses’ connections.

For teams that are on the old provisioning method some adjustments are required to their Helm charts or other provisioning code to switch to the new API. The old methods are still available but are considered deprecated and will be removed or break in the future.

AWS and Glue Connection provisioning [breaking]

With Lenses 5.4 IAM support was added for the AWS connection type. An AWS connection is used as an authentication provider for the Glue Schema Registry and Cloudwatch channels.

Due to this change, if you create or manage your AWS and Glue connections via the API or provisioning, you need to update your configuration to the new format.

Action required

• Add the new authMode property to your connections for AWS and Glue Schema Registry.

Details

• Lenses 5.4 adds a new required property for the AWS and Glue Schema Registry connections.

• The property is authMode.

• It controls how Lenses authenticates with AWS:

  • Access keys (existing feature).

  • Credentials provider chain (new feature).

• You set the property either with the:

  • Connections API - create, update.

  • Provision YAML.

You can set authMode in 2 modes:

1. Access keys mode

This is the existing mode where Lenses uses AWS access keys.

• Set the authMode to Access Key.

• Specify the access key ID and secret access key, as you had before.

{
  "authMode": { "value": "Access Key" },
  "accessKeyId": { "value": "yourAccessKeyId" },
  "secretAccessKey": { "value": "yourSecretAccessKey" }
}

2. Credentials provider chain mode (new)

This is the new mode where Lenses uses the AWS default credentials provider chain.

• Set the authMode to Credentials Chain.

• No additional properties needed.

{
  "authMode": { "value": "Credentials Chain" }
}

Examples - Provision YAML

1. Access mode

aws:
  - name: my-aws-connection
    version: 1
    tags: [dev]
    configuration:
      authMode:
        value: Access Key
      accessKeyId:
        value: yourAccessKeyId
      secretAccessKey:
        value: yourSecretAccessKey

glueSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: [dev]
    configuration:
      authMode:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:account:registry/registry-name
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection

2. Credentials provider chain mode

aws:
  - name: my-aws-connection
    version: 1
    tags: [dev]
    configuration:
      authMode: 
        value: Credentials Chain

glueSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: [dev]
    configuration:
      authMode:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:account:registry/registry-name

Examples - API JSON

1. Access mode

AWS connection

{
 "name": "my-aws-connection",
 "tags": ["dev"],
 "templateName": "AWS",
 "configuration": {
   "authMode": { "value": "Access Key" },
   "accessKeyId": { "value": "yourAccessKeyId" },
   "secretAccessKey": { "value": "yourSecretAccessKey" }
 }
}

Glue Schema Registry connection

{
    "name":"schema-registry",
    "tags": ["dev"],
    "templateName":"AWSGlueSchemaRegistry",
    "configuration": {
        "authMode": {"reference":"my-aws-connection"},
        "accessKeyId": {"reference":"my-aws-connection"},
        "secretAccessKey": {"reference":"my-aws-connection"},
        "glueRegistryArn":{"value":"arn:aws:glue:region:account:registry/registry-name"}
    }
}

2. Credentials provider chain mode

AWS connection

{
  "name": "my-aws-connection",
  "tags": ["dev"],
  "templateName": "AWS",
  "configuration": {
    "authMode": { "value": "Credentials Chain" }
  }
}

Glue Schema Registry connection

{
    "name":"schema-registry",
    "tags": ["dev"],
    "templateName":"AWSGlueSchemaRegistry",
    "configuration": {
        "authMode": {"reference":"my-aws-connection"},
        "glueRegistryArn":{"value":"arn:aws:glue:region:account:registry/registry-name"}
    }
}

Docker image base change

Starting with Lenses 5.2 the base image of Lenses and SQL Processor Dockers switched from Debian to Ubuntu. On some older systems, these docker images will fail to run, due to a combination of a recent glibc in the container, and older docker daemon on the host.

If you fall under this category, during the startup of the Lenses container, you might see errors such as Unable to identify system. Uname is required or [warning][os,thread] Failed to start thread “GC Thread#0”.

For these cases, we now offer Lenses docker images with the suffix -debian in their tags. E.g:

• lensesio/lenses:5.5-debian

• lensesio/lenses:5.5.0-debian

• lensesio/lenses:latest-debian

If your host is running on an older operating system and you encounter these errors, try to use the debian equivalent tag.

Update Process

Using the Lenses Archive

Download the latest 5.5 archive and extract it in a new directory on your server. It is important to avoid extracting an archive over an older installation to avoid having multiple versions of libraries. Instead, you should remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files. If you are using the internal database instead of PostgreSQL, make sure Lenses Storage Directory (lenses.storage.directory) is kept intact. The folder is where persistent data is stored, such as users, groups, audits, data policies, connections, and more.

Make sure you have a JRE (or JDK) installed in the server running Lenses. Lenses can run on JRE 8 or greater, and the recommended version is JRE 11.

Using the Lenses Docker

The docker image uses tags to distinguish between versions. The latest tag (lensesio/lenses:latest) brings the latest stable version of Lenses. There are minor tags to help users get the latest patch in a minor version (e.g 5.5, 5.1) and patch tags to help users pin to a specific patch (e.g 5.5.1, 5.1.2). The best practice advice is to use the minor tag (lensesio/lenses:5.5), which ensures that your installation will always get compatible updates until you made a conscious decision to upgrade the minor version.

If you use the internal database instead of PostgreSQL as the backing store of Lenses, make sure you keep the /data/storage volume to not lose your data. Other volumes supported by the docker are /data/kafka-streams-state which holds state for SQL Processors running IN-PROC and may have to be rebuilt (automatically) if lost, /data/log (log files on disk), /data/plugins (custom UDFs).

Pull the 5.5 docker:

docker pull lensesio/lenses:5.5

Stop your current container and restart with the 5.5 image, mounting any volumes you might need.

Lenses Box

If you are a Box user, pull the latest version, preserve your /data volume and restart Lenses:

docker pull lensesio/box:5.5
docker stop [CURRENT BOX NAME or ID]
docker run -v /path/to/box/data:/data -e EULA="..." -p 3030 lensesio/box:5.5

Helm

Download the latest charts and update your values.yaml as described below. Remember that if you are using the internal database instead of PostgreSQL as the backing store, then the Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates. To support a potential downgrade, make sure this volume is backed-up before installing a newer version of Lenses.

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

If you have provisioning enabled (lenses.provision.enabled: true) in your values.yaml, and you are on provision version “1” then you have to act. Version “1” means either that lenses.provision.version is set to "1", or it is not set at all. You have two options:

• Disable it, as Lenses already has all the information stored in the database, and version “1” does not support updating the connections and license.Copy

  Copy

  lenses:
    provision:
      enabled: false

• Switch to provisioning version “2” which supports updating connections and licenses every time you do a helm upgrade. To do that, you must make some changes to your old provisioning section. Some resources that can come handy for the switch are:

  • Provisioning API introduction

  • Provisioning API OpenAPI reference

  • Helm chart examples

If you don’t have your values.yaml you can download it from the Kubernetes cluster using Helm:

helm get values --namespace [LENSES_NAMESPACE] \
     --output yaml [LENSES_DEPLOYMENT] > values.yaml

Proceed to upgrade:

helm upgrade --namespace [LENSES_NAMESPACE] --values values.yaml [LENSES_DEPLOYMENT]

Alternatively, reusing the old values and turning provisioning off:

helm upgrade --namespace [LENSES_NAMESPACE] --reuse-values \
     --set lenses.provision.enabled=false [LENSES_DEPLOYMENT]

Cloud Installations

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates. If a new image does not exist, you may be able to update Lenses in-place. Our support team will be happy to go through the available options with you.

\