4.0
Upgrade notes
Lenses 4.0 is a major release which brings in new features to core components. Some of these are breaking changes, which means extra care should be taken before upgrading and following best practices: first upgrade development and staging environments, before moving to production.
Breaking Changes and Caution Items
Here is a shortlist of the changes that require actions before the upgrade. The most significant change is the new SQL Streaming, which affects SQL Processors.
- The first time Lenses 4.0 is run, it will upgrade the internal database. The process is not reversible. It is recommended to backup the internal database before upgrading, so it is possible to downgrade if needed. The database holds the user and service accounts, groups information, audits, alerts, and more.
- The new SQL Streaming engine now uses the same dialect as the SQL snapshot engine
(used for data exploration), which makes queries written for the old SQL Streaming
incompatible. Furthermore the deployment procedure has been rewritten and made
more robust. As such:
- SQL Processors running with IN_PROC mode will be lost after the upgrade. A lenses-cli release for the 3.2 series will follow soon to allow exporting IN_PROC processors.
- SQL Processors CONNECT will skip 4.0. This mode is expected to return with Lenses 4.1. Any processors created with Lenses 3.2 or older, will keep running in Kafka Connect but will not be visible in Lenses. They will show up again in 4.1. At that time if you upgrade your SQL Connector to 4.1, you will have to rewrite your old processors.
- SQL Processors in KUBERNETES created in Lenses 3.2 or older, will not be visible in Lenses 4.0. They will show up again in Lenses 4.1. You will be able to create new processors in Kubernetes, using the new SQL Streaming.
- The User Defined Functions (UDF and UDAF) got SQL Streaming support, but due to this upgrade the programming interfaces have changed, so existing UDFs must be updated.
- Custom Serializers and Deserializers should be rebuilt using the latest version of the lenses-serde library.
Operational changes:
The SQL Processor docker image has been made public, which means you don’t have to patch your Kubernetes namespaces anymore. Starting with Lenses 4.0 and later all images are public. Your Kubernetes cluster should allow pulling images from Docker Hub. There are two images needed for using SQL Streaming in Kubernetes:
lensesio/lenses-cli
lensesioextra/sql-processor
The license can be updated with one of the following methods:
- Update the license file while Lenses is running
- Use the lenses-cli or a REST call (e.g., curl) to update the license while Lenses is running.
It is not possible to update the license when Lenses is not running. Changes made to the license file will be ignored.
Update Process
Please backup your database (lenses.storage.directory
) before updating to
Lenses 4.0. Downgrading from 4.0 to 3.2 is not supported.
Using the Lenses Archive
Download the
latest 4.0 archive
and extract it in a new directory on your server. It is important to avoid
extracting an archive over an older installation to avoid having multiple
versions of libraries. Instead, you should remove (or rename) the old directory,
then move the new into its place. Copy if needed and update your lenses.conf
and security.conf
files as described later - see(Configuration Changes).
Make sure Lenses Storage Directory is kept intact. The folder is where persistent
data is stored, such as users, groups, audits, data policies, connections, and more.
Using the Lenses Docker
The docker image uses tags to distinguish between versions. The latest
tag
(lensesio/lenses:latest
) brings the latest stable version of Lenses. There are
minor tags to help users get the latest patch in a minor version (e.g 4.0
,
3.2
) and patch tags to help users pin to a specific patch (e.g 4.0.0
,
3.2.5
). The best practice advice is to use the minor tag
(lensesio/lenses:4.0
), which ensures that your installation will always get
compatible updates until you made a conscious decision to upgrade the minor
version.
To get Lenses 4.0, make sure you keep at least your /data/storage
volume
to not lose your data. Other volumes supported by the docker are
/data/kafka-streams-state
(SQL Processors state), /data/log
(log files on
disk), /data/plugins
(custom UDFs).
Pull the 4.0 docker:
docker pull lensesio/lenses:4.0
Follow the Configuration Changes section to update your settings if needed. Stop
your current container and restart with the 4.0
image, mounting any volumes you might need.
Lenses Box
If you are a Box user, pull the latest version, preserve your /data
volume and restart Lenses:
docker pull lensesio/box:4.0
docker stop [CURRENT BOX NAME or ID]
docker run -v /path/to/box/data:/data -e EULA="..." -p 3030 lensesio/box:4.0
Helm
Download the latest charts and update your values.yaml
as described in
Configuration Changes. Remember that Lenses Storage Directory should be stored
in a persistent volume and be kept intact between updates. To support a potential
downgrade, make sure this volume is backed-up before installing a newer version of Lenses.
helm repo add lensesio https://helm.repo.lenses.io/
helm repo update
Cloud Installations
Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates.
SQL Processors
The SQL Streaming dialect has changes. Existing queries will require some simple rewrite in order to work.
IN_PROC processors will be lost during the upgrade. A new version of the lenses-cli
will soon be available to allow exporting of IN_PROC processors. Then you can edit your export files and import them to Lenses 4.0.
CONNECT processors are not supported in Lenses 4.0 but will come back with Lenses 4.1. Lenses 4.0 will not start, as a non-compatible upgrade path has been detected. Currently deployed processors will keep working but not show in Lenses. They will start showing again in Lenses 4.1, but once you upgrade the SQL Connector, you will have to rewrite your processors for the new dialect.
KUBERNETES processors created with versions prior to 4.0 they will not show in Lenses 4.0. They will keep working and show up again with Lenses 4.1.
Configuration
Configuration wise this version of Lenses is compatible with 3.2. A few configuration keys have been deprecated. Ideally, they should be removed, though their presence will not affect the upgrade process.
Below you will also find a list of configuration keys that have changed. They are rarely used, and most users will not have to take any action.
Changes
The following configuration keys have been renamed:
Old config | New config |
---|---|
lenses.interval.jmx.refresh.zk | lenses.interval.metrics.refresh.zk |
lenses.interval.jmx.refresh.sr | lenses.interval.metrics.refresh.sr |
lenses.interval.jmx.refresh.broker | lenses.interval.metrics.refresh.broker |
lenses.interval.jmx.refresh.connect | lenses.interval.metrics.refresh.connect |
lenses.interval.jmx.refresh.brokers.in.zk | lenses.interval.metrics.refresh.brokers.in.zk |
lenses.interval.jmx.refresh.alert.manager | lenses.interval.metrics.refresh.alert.manager |
Streaming SQL on Kubernetes changes
Starting with this version the following configurations have been dropped. The Kafka connection is now plugged in automatically:
Deprecated config |
---|
lenses.kubernetes.processor.kafka.* |
lenses.kubernetes.processor.schema.registry.* |
lenses.kubernetes.processor.jaas |
lenses.kubernetes.processor.krb5 |
lenses.kubernetes.runner.java.opts |
lenses.kubernetes.processor.heap |
lenses.kubernetes.processor.mem.request |
lenses.kubernetes.processor.mem.limit |
The following configuration keys have been renamed:
Old config | New config |
---|---|
lenses.kubernetes.runner.mem.limit | lenses.kubernetes.pod.mem.limit |
lenses.kubernetes.runner.mem.request | lenses.kubernetes.pod.mem.request |
CLI
The command line interface has the following changes:
Old command | New command | Example |
---|---|---|
resume | start | ./lenses-cli processor start … |
pause | stop | ./lenses-cli processor stop … |
The YAML export and import format for processors has changed.
Exporting processors YAMLs in Lenses 3.2 and importing to this version requires manual changes.
APIs
See the API documentation
Alerts
The following APIs have been deprecated. Find the alternative APIs bellow:
Deprecated Legacy API | New API |
---|---|
GET /api/alerts | GET /api/v1/alert/events |
DELETE /api/alerts | DELETE /api/v1/alert/events |
GET /api/alerts/settings | GET /api/v1/alert/settings |
GET /api/alerts/settings | GET /api/v1/alert/settings |
PUT /api/alerts/settings/{sid} | PUT /api/v1/alert/settings/{sid} |
POST /api/alerts/settings/{sid}/condition | POST /api/v1/alert/settings/{sid}/conditions |
DELETE /api/alerts/settings/{sid}/condition/{cid} | POST /api/v1/alert/settings/{sid}/conditions/{cid} |
GET /api/alerts/integrations | GET /api/v1/alert/channels |
Connections and templates
The following APIs expect DOUBLE rather than NUMBER as the enum type for parameter ’type.name'.
Affected API |
---|
GET /api/v1/connection/connection-templates |
GET /api/v1/alert/channel-templates |