Upgrade notes

Lenses 4.1 is an incremental release for the 4.x series which brings in new features and improvements to core components.

Changes in the configuration settings may require the Lenses administrator to perform adjustments to lenses.conf during the upgrade. Users upgrading from Lenses 3.2 need to pay further attention as there are breaking changes for SQL Processors.

Following industry’s best practices, it is recommended to review the list of important changes, backup Lenses internal database, and if development and staging environments are available, to upgrade them first before moving to production.

Important Changes and Caution Items

From Lenses 4.0

Connect Clusters might have to be renamed in lenses.conf. Connect cluster names do not support underscores (_) anymore, but support for dashes (-) was added.
Connect clusters can not longer be configured with lenses.connect.clusters configuration key in lenses.conf. lenses.kafka.connect.clusters key should be used instead. Check Kafka Connect Configurtion for more information about Connect clusters configuration.
SASL JAAS configuration for Kafka Brokers standardized to inline in lenses.conf. External jaas.conf files will keep working for the time being, but going forward inline configuration is the only supported method. Please read Kafka Brokers Configuration for more information.
Schema Registry authentication configuration standardized. For Basic Auth to the Schema Registry a few entries in lenses.conf might need to be removed. TLS Authentication is officially supported and if you currently use workarounds in lenses.conf, you have to remove them and use the suggested settings. Please read Schema Registry Configuration for more information.
SQL Processor CONNECT mode is available again. To create new processors Kafka Connect 2.5 is required, as well as the new SQL Connector and Secrets Provider plugin. Check Connect SQL Streaming for more information.
SQL Processors in CONNECT mode from Lenses 3.2 will re-appear. If you have SQL Processors running in your Connect cluster they will appear in the Processors screen. These processors are deprecated and it is advised to delete and recreate them if possible, to take advantage of the new SQL Streaming and the more robust new deployment framework. Old processors will have a simplified topology view until upgraded. For the import to work correctly please make sure:
- Connect clusters are accessible and healthy at the time Lenses starts. If the Connect clusters are down, importing the old processors can still happen later in time, but it requires a Lenses restart.
- Processors stopped in Lenses are not going to be imported. Make sure all the processors required to be imported are running at the time Lenses 4.1 starts.
SQL Processors in KUBERNETES mode from Lenses 4.0 will continue working as expected. SQL Processors in KUBERNETES mode from Lenses 3.2 will not show in Lenses 4.1.0 but will keep working. With the next patch release of Lenses 4.1.1 they will be imported automatically in the new deployment framework and appear as deprecated in the UI —though you will be able to manage them. It is advised to delete and recreate them, if possible, to take advantage of the new SQL Streaming and the full capabilities of the new, more robust deployment framework. They will also have a simplified topology view until upgraded.
The internal database structure has changed. During the first run of 4.1 a migration will run. The process is not reversible. It is recommended to backup the internal database before upgrading, so it is possible to downgrade if needed. The database holds the user and service accounts, groups information, audits, alerts, and more.

From Lenses 3.2

Please also check the upgrade guide from 3.2 to 4.0 , especially for API changes.

The internal database structure has changed. During the first run of 4.1 a migration will run. The process is not reversible. It is recommended to backup the internal database before upgrading, so it is possible to downgrade if needed. The database holds the user and service accounts, groups information, audits, alerts, and more.
SQL Processors running in IN_PROC mode will be lost during the upgrade. You can use lenses-cli export processors to export them. Before you import them you will need to edit the SQL queries for the new SQL Streaming.
To create new SQL Processors in CONNECT mode requires Kafka Connect 2.5 as well as the new SQL Connector and Secrets Provider plugin. Check Connect SQL Streaming for more information.
SQL Processors running in CONNECT mode from Lenses 3.2 will be imported automatically in the new deployment framework and appear as deprecated in the UI —though you will be able to manage them. It is advised to delete and recreate them, if possible, to take advantage of the new SQL Streaming and the full capabilities of the new, more robust deployment framework. They will also have a simplified topology view until upgraded. For the import to work correctly please make sure:
- Connect clusters are accessible and healthy at the time Lenses starts. If the Connect clusters are down, importing the old processors can still happen later in time, but it requires a Lenses restart.
- Processors stopped in Lenses are not going to be imported. Make sure all the processors required to be imported are running at the time Lenses 4.1 starts.
SQL Processors in KUBERNETES mode from Lenses 3.2 will not show in Lenses 4.1.0 but will keep working. With the next patch release of Lenses 4.1.1 they will be imported automatically in the new deployment framework and appear as deprecated in the UI —though you will be able to manage them. It is advised to delete and recreate them, if possible, to take advantage of the new SQL Streaming and the full capabilities of the new, more robust deployment framework. They will also have a simplified topology view until upgraded.
Connect Clusters might have to be renamed in lenses.conf. Connect cluster names do not support underscores (_) anymore, but support for dashes (-) was added.
Connect clusters can not longer be configured with lenses.connect.clusters configuration key in lenses.conf. lenses.kafka.connect.clusters key should be used instead. Check Kafka Connect Configurtion for more information about Connect clusters configuration.
SASL JAAS configuration for Kafka Brokers standardized to inline in lenses.conf. External jaas.conf files will keep working for the time being, but going forward inline configuration is the only supported method. Please read Kafka Brokers Configuration for more information.
Schema Registry authentication configuration standardized. For Basic Auth to the Schema Registry a few entries in lenses.conf might need to be removed. TLS Authentication is officially supported and if you currently use workarounds in lenses.conf, you have to remove them and use the suggested settings. Please read Schema Registry Configuration for more information.
User-Defined Functions (UDF and UDAF) are now supported by the SQL Streaming engine. Due to this upgrade and changes made, existing UDFs have to be updated or rebuilt.

Operational changes:

The SQL Processor docker image has been made public, which means you don’t have to patch your Kubernetes namespaces anymore. Starting with Lenses 4.0 and later all images are public. Your Kubernetes cluster should allow pulling images from Docker Hub. There are two images needed for using SQL Streaming in Kubernetes:
- lensesio/lenses-cli
- lensesioextra/sql-processor
The license can be updated with one of the following methods:
- Update the license from Lenses UI Admin section
- Update the license file while Lenses is running
- Use the lenses-cli or a REST call (e.g., curl) to update the license while Lenses is running It is not possible to update the license when Lenses is not running. Changes made to the license file will be ignored.

Update Process

Please backup your database (lenses.storage.directory) before updating to Lenses 4.1. The backup method depends on your deployment type.

Using the Lenses Archive

Download the latest 4.1 archive and extract it in a new directory on your server. It is important to avoid extracting an archive over an older installation to avoid having multiple versions of libraries. Instead, you should remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files as described later - see (Configuration Changes). Make sure Lenses Storage Directory is kept intact. The folder is where persistent data is stored, such as users, groups, audits, data policies, connections, and more.

Using the Lenses Docker

The docker image uses tags to distinguish between versions. The latest tag (lensesio/lenses:latest) brings the latest stable version of Lenses. There are minor tags to help users get the latest patch in a minor version (e.g. 4.1, 3.2) and patch tags to help users pin to a specific patch (e.g. 4.1.0, 3.2.10). The best practice advice is to use the minor tag (lensesio/lenses:4.1), which ensures that your installation will always get compatible updates until you made a conscious decision to upgrade the minor version.

To get Lenses 4.1, make sure you keep at least your /data/storage volume to not lose your data. Other volumes supported by the docker are /data/kafka-streams-state (SQL Processors state), /data/log (log files on disk), /data/plugins (custom UDFs).

Pull the 4.1 docker:

docker pull lensesio/lenses:4.1

Follow the Configuration Changes section to update your settings if needed. Stop your current container and restart with the 4.1 image, mounting any volumes you might need.

Lenses Box

If you are a Box user, pull the latest version, preserve your /data volume and restart Lenses.

docker pull lensesio/box:4.1
docker stop [CURRENT BOX NAME or ID]
docker run -v /path/to/box/data:/data -e EULA="..." -p 3030 lensesio/box:4.1

Helm

Download the latest charts and update your values.yaml as described in Configuration Changes. Remember that Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates. To support a potential downgrade, make sure this volume is backed-up before installing a newer version of Lenses.

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

Changes

SASL JAAS configuration for Kafka Brokers standardized to inline in lenses.conf. Consequently, Helm chart ‘values.yaml’ should be also changed. Please read Kafka Brokers Configuration for more information.
- jaasConfig was introduced to set JAAS content inline instead of using a file, remember to not include KafkaClient{} wrapping
```
# values.yaml
lenses:
  kafka:
    sasl:
      enabled: true
      jaasConfig: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="lenses.keytab" storeKey=true useTicketCache=false serviceName=kafka principal="lenses@TESTING.LENSES.IO";
```
- jaasFileData was retained for backward compatibilty reasons but it is marked as deprecated and will be removed in future version.
schemaRegistries.enabled is by default set to false, to activate Schema registry integration you need to explicitly enable it.
connectClusters.enabled is by default set to false, to activate Connect clusters integration you need to explicitly enable it.

Cloud Installations

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates.

Configuration

Configuration wise this version of Lenses is mostly compatible with 4.0.

Connect clusters that use underscores (_) in their names should be edited to remove them. Dash (-) support was added, so you may choose to replace underscores with dashes.

The SASL JAAS configuration for Kafka Brokers has standardized to inline. External jaas.conf files will keep working for the time being, but going forward inline configuration is the only supported method. Please read Kafka Brokers Configuration for more information.

Schema Registry authentication configuration has been standardized. For Basic Auth to the Schema Registry a few configuration entries might need to be removed. TLS Authentication is officially supported and if you currently use workarounds in lenses.conf, you have to remove them and use the suggested settings. Please read Schema Registry Configuration for more information.

APIs

See the API documentation

Kafka Connect Cluster permissions

After migrating to Lenses 4.1, all the existing groups automatically will have permissions to all the configured connect clusters.