4.0

Upgrade notes

Lenses 4.0 is a major release which brings in new features to core components. Some of these are breaking changes, which means extra care should be taken before upgrading and following best practices: first upgrade development and staging environments, before moving to production.

Breaking Changes and Caution Items 

Here is a shortlist of the changes that require actions before the upgrade. The most significant change is the new SQL Streaming, which affects SQL Processors.

  • The first time Lenses 4.0 is run, it will upgrade the internal database. The process is not reversible. It is recommended to backup the internal database before upgrading, so it is possible to downgrade if needed. The database holds the user and service accounts, groups information, audits, alerts, and more.
  • The new SQL Streaming engine now uses the same dialect as the SQL snapshot engine (used for data exploration), which makes queries written for the old SQL Streaming incompatible. Furthermore the deployment procedure has been rewritten and made more robust. As such:
    • SQL Processors running with IN_PROC mode will be lost after the upgrade. A lenses-cli release for the 3.2 series will follow soon to allow exporting IN_PROC processors.
    • SQL Processors CONNECT will skip 4.0. This mode is expected to return with Lenses 4.1. Any processors created with Lenses 3.2 or older, will keep running in Kafka Connect but will not be visible in Lenses. They will show up again in 4.1. At that time if you upgrade your SQL Connector to 4.1, you will have to rewrite your old processors.
    • SQL Processors in KUBERNETES created in Lenses 3.2 or older, will not be visible in Lenses 4.0. They will show up again in Lenses 4.1. You will be able to create new processors in Kubernetes, using the new SQL Streaming.
  • The User Defined Functions (UDF and UDAF) got SQL Streaming support, but due to this upgrade the programming interfaces have changed, so existing UDFs must be updated.
  • Custom Serializers and Deserializers should be rebuilt using the latest version of the lenses-serde library.

Operational changes:

  • The SQL Processor docker image has been made public, which means you don’t have to patch your Kubernetes namespaces anymore. Starting with Lenses 4.0 and later all images are public. Your Kubernetes cluster should allow pulling images from Docker Hub. There are two images needed for using SQL Streaming in Kubernetes:

    • lensesio/lenses-cli
    • lensesioextra/sql-processor
  • The license can be updated with one of the following methods:

    • Update the license file while Lenses is running
    • Use the lenses-cli or a REST call (e.g., curl) to update the license while Lenses is running.

    It is not possible to update the license when Lenses is not running. Changes made to the license file will be ignored.

Update Process 

Please backup your database (lenses.storage.directory) before updating to Lenses 4.0. Downgrading from 4.0 to 3.2 is not supported.

Using the Lenses Archive 

Download the latest 4.0 archive and extract it in a new directory on your server. It is important to avoid extracting an archive over an older installation to avoid having multiple versions of libraries. Instead, you should remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files as described later - see(Configuration Changes). Make sure Lenses Storage Directory is kept intact. The folder is where persistent data is stored, such as users, groups, audits, data policies, connections, and more.

Using the Lenses Docker 

The docker image uses tags to distinguish between versions. The latest tag (lensesio/lenses:latest) brings the latest stable version of Lenses. There are minor tags to help users get the latest patch in a minor version (e.g 4.0, 3.2) and patch tags to help users pin to a specific patch (e.g 4.0.0, 3.2.5). The best practice advice is to use the minor tag (lensesio/lenses:4.0), which ensures that your installation will always get compatible updates until you made a conscious decision to upgrade the minor version.

To get Lenses 4.0, make sure you keep at least your /data/storage volume to not lose your data. Other volumes supported by the docker are /data/kafka-streams-state (SQL Processors state), /data/log (log files on disk), /data/plugins (custom UDFs).

Pull the 4.0 docker:

docker pull lensesio/lenses:4.0

Follow the Configuration Changes section to update your settings if needed. Stop your current container and restart with the 4.0 image, mounting any volumes you might need.

Lenses Box 

If you are a Box user, pull the latest version, preserve your /data volume and restart Lenses:

docker pull lensesio/box:4.0
docker stop [CURRENT BOX NAME or ID]
docker run -v /path/to/box/data:/data -e EULA="..." -p 3030 lensesio/box:4.0

Helm 

Download the latest charts and update your values.yaml as described in Configuration Changes. Remember that Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates. To support a potential downgrade, make sure this volume is backed-up before installing a newer version of Lenses.

helm repo add lensesio https://helm.repo.lenses.io/
helm repo update

Cloud Installations 

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates.

SQL Processors 

The SQL Streaming dialect has changes. Existing queries will require some simple rewrite in order to work.

IN_PROC processors will be lost during the upgrade. A new version of the lenses-cli will soon be available to allow exporting of IN_PROC processors. Then you can edit your export files and import them to Lenses 4.0.

CONNECT processors are not supported in Lenses 4.0 but will come back with Lenses 4.1. Lenses 4.0 will not start, as a non-compatible upgrade path has been detected. Currently deployed processors will keep working but not show in Lenses. They will start showing again in Lenses 4.1, but once you upgrade the SQL Connector, you will have to rewrite your processors for the new dialect.

KUBERNETES processors created with versions prior to 4.0 they will not show in Lenses 4.0. They will keep working and show up again with Lenses 4.1.

Configuration 

Configuration wise this version of Lenses is compatible with 3.2. A few configuration keys have been deprecated. Ideally, they should be removed, though their presence will not affect the upgrade process.

Below you will also find a list of configuration keys that have changed. They are rarely used, and most users will not have to take any action.

Changes 

The following configuration keys have been renamed:

Old configNew config
lenses.interval.jmx.refresh.zk
lenses.interval.metrics.refresh.zk
lenses.interval.jmx.refresh.sr
lenses.interval.metrics.refresh.sr
lenses.interval.jmx.refresh.broker
lenses.interval.metrics.refresh.broker
lenses.interval.jmx.refresh.connect
lenses.interval.metrics.refresh.connect
lenses.interval.jmx.refresh.brokers.in.zk
lenses.interval.metrics.refresh.brokers.in.zk
lenses.interval.jmx.refresh.alert.manager
lenses.interval.metrics.refresh.alert.manager

Streaming SQL on Kubernetes changes 

Starting with this version the following configurations have been dropped. The Kafka connection is now plugged in automatically:

Deprecated config
lenses.kubernetes.processor.kafka.*
lenses.kubernetes.processor.schema.registry.*
lenses.kubernetes.processor.jaas
lenses.kubernetes.processor.krb5
lenses.kubernetes.runner.java.opts
lenses.kubernetes.processor.heap
lenses.kubernetes.processor.mem.request
lenses.kubernetes.processor.mem.limit

The following configuration keys have been renamed:

Old configNew config
lenses.kubernetes.runner.mem.limit
lenses.kubernetes.pod.mem.limit
lenses.kubernetes.runner.mem.request
lenses.kubernetes.pod.mem.request

CLI 

The command line interface has the following changes:

Old commandNew commandExample
resume
start
./lenses-cli processor start …
pause
stop
./lenses-cli processor stop …

The YAML export and import format for processors has changed.

Exporting processors YAMLs in Lenses 3.2 and importing to this version requires manual changes.

APIs 

See the API documentation

Alerts 

The following APIs have been deprecated. Find the alternative APIs bellow:

Deprecated Legacy APINew API
GET /api/alerts
GET /api/v1/alert/events
DELETE /api/alerts
DELETE /api/v1/alert/events
GET /api/alerts/settings
GET /api/v1/alert/settings
GET /api/alerts/settings
GET /api/v1/alert/settings
PUT /api/alerts/settings/{sid}
PUT /api/v1/alert/settings/{sid}
POST /api/alerts/settings/{sid}/condition
POST /api/v1/alert/settings/{sid}/conditions
DELETE /api/alerts/settings/{sid}/condition/{cid}
POST /api/v1/alert/settings/{sid}/conditions/{cid}
GET /api/alerts/integrations
GET /api/v1/alert/channels

Connections and templates 

The following APIs expect DOUBLE rather than NUMBER as the enum type for parameter ’type.name'.

Affected API
GET /api/v1/connection/connection-templates
GET /api/v1/alert/channel-templates