4.3

Upgrade notes

Lenses 4.3 is an incremental release for the 4.x series which brings in new features and improvements to core components.

Users upgrading from 4.1 need to pay attention for breaking changes to the SQL Processors running IN_PROC. Users coming from 4.0, in addition may need to perform adjustments to lenses.conf. Users upgrading from 3.2 need to pay further attention as there are breaking changes for SQL Processors in all modes.

Following industry’s best practices, it is recommended to review the list of important changes, backup Lenses internal database, and if development and staging environments are available, to upgrade them first before moving to production.

Important Changes and Caution Items 

Upgrading from Lenses 4.2 

There is no breaking change, nor any configuration change administrators need to attend to.

Upgrading from Lenses 4.1 

  • SQL Processors running on IN_PROC

    • Please note that IN_PROC mode purpose is for testing the SQL Processor functionality. Once you are ready you should graduate to production deployment modes (Kubernetes or Kafka Connect).

    • During this upgrade SQL Processors running IN_PROC will be lost. You may export them before the upgrade, and import them afterwards.

    • Anytime Lenses is restarted, existing SQL Processors running IN_PROC will switch to the stopped (NOT RUNNING) state.

    • SQL Processors running IN_PROC switched to the Connections Management framework, like the KUBERNETES and CONNECT modes before them. Learn more about IN_PROC and connection management .

    • SQL Processors running IN_PROC are limited to 50 in order to prevent resource exhaustion.

  • SQL Processors running on CONNECT

    • The use of secret provider is mandatory for Connect clusters that run the SQL Connector 4.x series. Connect clusters without the plugin will not be listed as an option anymore when creating a SQL Processor. Learn how to setup the secret provider .
  • To take advantage of the new Datasets’ Tags and Description functionality your user groups require a new permission (UpdateMetadata). Please refer to our Help Center Metadata Security section for more information.

Upgrading from Lenses 4.0 

  • All changes from Lenses 4.1 apply.

  • Connect Clusters might have to be renamed in lenses.conf. Connect cluster names do not support underscores (_) anymore, but support for dashes (-) was added.

  • Connect clusters can no longer be configured with lenses.connect.clusters configuration key in lenses.conf. lenses.kafka.connect.clusters key should be used instead. Check Kafka Connect Configuration for more information about Connect clusters configuration.

  • SASL JAAS configuration for Kafka Brokers standardized to inline in lenses.conf. External jaas.conf files will keep working for the time being, but going forward inline configuration is the only supported method. Please read Kafka Brokers Configuration for more information.

  • Schema Registry authentication configuration standardized. For Basic Auth to the Schema Registry a few entries in lenses.conf might need to be removed. TLS Authentication is officially supported and if you currently use workarounds in lenses.conf, you have to remove them and use the suggested settings. Please read Schema Registry Configuration for more information.

  • SQL Processors in CONNECT mode from Lenses 3.2 will re-appear. The CONNECT mode was skipped for Lenses 4.0 but was brought back with version 4.1. If you have SQL Processors running in your Connect cluster they will appear in the Processors screen. These processors are deprecated and it is advised to delete and recreate them if possible, to take advantage of the new SQL Streaming and the more robust new deployment framework. Old processors will have a simplified topology view until upgraded. For the import to work correctly please make sure:

    • Connect clusters are accessible and healthy at the time Lenses starts. If the Connect clusters are down, importing the old processors can still happen later in time, but it requires a Lenses restart.
    • Processors stopped in Lenses are not going to be imported. Make sure all the processors required to be imported are running at the time Lenses 4.1 starts.
  • SQL Processors in KUBERNETES mode from Lenses 4.0 will continue working as expected. SQL Processors in KUBERNETES mode from Lenses 3.2 will appear as deprecated in the UI —though you will be able to manage them. It is advised to delete and recreate them, if possible, to take advantage of the new SQL Streaming and the full capabilities of the new, more robust deployment framework. They will also have a simplified topology view until upgraded.

  • The internal database structure has changed. During the first run of 4.2 a migration will run. The process is not reversible. It is recommended to backup the internal database before upgrading, so it is possible to downgrade if needed. The database holds the user and service accounts, groups information, audits, alerts, and more.

  • Connect clusters now have permissions. Existing user groups will get automatically permissions to all existing Connect clusters in order to not alter the behaviour before and after the upgrade. You can change these permissions within each group’s configuration. For new clusters and groups, explicit access permission is required..

Upgrading from Lenses 3.2 

Please also check the upgrade guide from 3.2 to 4.0 , especially for API changes.

  • All changes from Lenses 4.1 apply.

  • The internal database structure has changed. During the first run of 4.2 a migration will run. The process is not reversible. It is recommended to backup the internal database before upgrading, so it is possible to downgrade if needed. The database holds the user and service accounts, groups information, audits, alerts, and more.

  • SQL Processors running in IN_PROC mode will be lost during the upgrade. You can use lenses-cli export processors to export them. Before you import them you will need to edit the SQL queries for the new SQL Streaming.

  • SQL Processors running in CONNECT mode will be imported automatically in the new deployment framework and appear as deprecated in the UI —though you will be able to manage them. It is advised to delete and recreate them, if possible, to take advantage of the new SQL Streaming and the full capabilities of the new, more robust deployment framework. They will also have a simplified topology view until upgraded. For the import to work correctly please make sure:

    • Connect clusters are accessible and healthy at the time Lenses starts. If the Connect clusters are down, importing the old processors can still happen later in time, but it requires a Lenses restart.
    • Processors stopped in Lenses are not going to be imported. Make sure all the processors required to be imported are running at the time Lenses 4.2 starts.
  • Connect Clusters might have to be renamed in lenses.conf. Connect cluster names do not support underscores (_) anymore, but support for dashes (-) was added.

  • Connect clusters can not longer be configured with lenses.connect.clusters configuration key in lenses.conf. lenses.kafka.connect.clusters key should be used instead. Check Kafka Connect Configurtion for more information about Connect clusters configuration.

  • SASL JAAS configuration for Kafka Brokers standardized to inline in lenses.conf. External jaas.conf files will keep working for the time being, but going forward inline configuration is the only supported method. Please read Kafka Brokers Configuration for more information.

  • Schema Registry authentication configuration standardized. For Basic Auth to the Schema Registry a few entries in lenses.conf might need to be removed. TLS Authentication is officially supported and if you currently use workarounds in lenses.conf, you have to remove them and use the suggested settings. Please read Schema Registry Configuration for more information.

  • User-Defined Functions (UDF and UDAF) are now supported by the SQL Streaming engine. Due to this upgrade and changes made, existing UDFs have to be updated or rebuilt.

  • Connect clusters now have permissions. Existing user groups will get automatically permissions to all existing Connect clusters in order to not alter the behaviour before and after the upgrade. You can change these permissions within each group’s configuration. For new clusters and groups, explicit access permission is required..

Operational changes:

  • The SQL Processor docker image has been made public, which means you don’t have to patch your Kubernetes namespaces anymore. Starting with Lenses 4.0 and later all images are public. Your Kubernetes cluster should allow pulling images from Docker Hub. There are two images needed for using SQL Streaming in Kubernetes:
    • lensesio/lenses-cli
    • lensesioextra/sql-processor
  • The license can be updated with one of the following methods:
    • Update the license from Lenses UI Admin section
    • Update the license file while Lenses is running
    • Use the lenses-cli or a REST call (e.g., curl) to update the license while Lenses is running It is not possible to update the license when Lenses is not running. Changes made to the license file will be ignored.

Update Process 

Please backup your database before updating to Lenses 4.3. If you are using local storage for the database you have to make a copy of the storage directory (lenses.storage.directory). The method depends on your deployment type (directory on local disk, docker volume, kubernetes persistent claim). If you are using Postgres, make an export of the Lenses database.

Using the Lenses Archive 

Download the latest 4.3 archive and extract it in a new directory on your server. It is important to avoid extracting an archive over an older installation to avoid having multiple versions of libraries. Instead, you should remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files as described earlier - see Configuration Changes . If using local storage for the database, make sure the Lenses Storage Directory is kept intact. The folder is where persistent data is stored, such as users, groups, audits, data policies, connections, and more.

Using the Lenses Docker 

The docker image uses tags to distinguish between versions. The latest tag (lensesio/lenses:latest) brings the latest stable version of Lenses. There are minor tags to help users get the latest patch in a minor version (e.g. 4.3, 3.2) and patch tags to help users pin to a specific patch (e.g. 4.3.0, 3.2.10). The best practice advice is to use the minor tag (lensesio/lenses:4.3), which ensures that your installation will always get compatible updates until you make a conscious decision to upgrade the minor version.

To get Lenses 4.3, make sure you keep at least your /data/storage volume to not lose your data. Other volumes supported by the docker are /data/kafka-streams-state (SQL Processors state), /data/log (log files on disk), /data/plugins (custom UDFs).

Pull the 4.3 docker:

docker pull lensesio/lenses:4.3

Follow the Configuration Changes section to update your settings if needed. Stop your current container and restart with the 4.3 image, mounting any volumes you might need.

Lenses Box 

If you are a Box user, pull the latest version, preserve your /data volume and restart Lenses.

docker pull lensesio/box:4.3
docker stop [CURRENT BOX NAME or ID]
docker run -v /path/to/box/data:/data -e EULA="..." -p 3030 lensesio/box:4.3

Helm 

Download the latest charts and update your values.yaml as described in Configuration Changes. Remember that Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates. To support a potential downgrade, make sure this volume is backed-up before installing a newer version of Lenses.

helm repo add lensesio https://helm.repo.lenses.io
helm repo update

Cloud Installations 

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates.