Upgrade Notes¶

Upgrading from an older version of Lenses, requires some changes to the existing configuration files. Please familiarize with the changes described below before performing an upgrade.

For the full list of changes made for this release please check the Release Notes.

Note

Please remember that Lenses 2.2 introduced state on disk for the Data Policies module. You have to keep your Lenses Storage Directory in order for your Data Policies to transfer to 3.0. Furthermore, in 3.0 user management, audits, and alerts have been migrated to the Lenses Storage Directory, so you must make sure you persist these data in order to be able to retain them between upgrades in the future.

From 2.3 to 3.0¶

Application Update¶

Archive¶

Download the latest archive and extract it in a new directory. It is important to avoid extracting an archive over an older installation. You should instead remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files as described later in the current page (Configuration Changes). Last make sure you keep the Lenses Storage Directory intact as this is where persistent data is stored. Up to Lenses 2.3 Data Policies are stored there. Starting with 3.0, users, groups, audits and other data is stored as well.

Docker¶

As usual the latest tag brings the latest Lenses versions. There are also tagged releases major (3.0, 2.3) and even minor (3.0.0, 2.3.6) tags to help you better manage your upgrade path.

Starting with 3.0, the landoop/kafka-lenses-dev image can also be accessed via lensesio/box. The landoop/lenses image can also be accessed via the lensesio/lenses name. In the future the old names will be deprecated, thus it’s advised to make the switch to the new images earlier rather than later.

Docker pull the latest Lenses environment for Developers:

docker pull lensesio/box:latest

Or update you docker container for Lenses Enterprise with:

docker pull lensesio/lenses:latest

Follow the Configuration Changes section to update your settings if needed. Remember that Lenses Storage Directory should be stored in a docker volume and be kept intact between updates.

Helm¶

Download the latest charts and update your values.yaml as described in Configuration Changes and Helm Charts Changes sections. Remember that Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates.

Cloud Installations¶

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates.

SQL Processors¶

For kubernetes no action is required. New processors will use automatically the latest image. Old processors will keep working with the old image.

For Kafka Connect, please download the latest version of the connector and add it to your connect nodes whilst removing the old version. Your nodes need to be restarted for the change to take effect. A rolling restart is advised.

Configuration Changes¶

User and Group management logic became dynamic and moved into the Lenses API. You can manage users and groups via the Lenses web interface or the lenses-cli application. What this means for operators, is that the security.conf file is no longer used to setup user acounts for the BASIC authentication module, neither groups or user-group mappings.
- If you are using the BASIC authentication module you must recreate your users via the web interface or the command line.
- You must recreate your groups via the Web Interface or the command line. This is also a great chance to familiarize with the new, fine grained, permission system.
- If you are using the Kerberos (SPNEGO) authentication module you must add your principals and link them to groups via the Web Interface or the command line.
- If you are using service accounts you must recreate them via the web interface or the command line.
Check User Management and Security Configuration to familiarize with the changes.

Note

This version comes with a default administrator user. Remember to secure your setup with a custom username and password for this user. Use SHA256 to protect the password inside security.conf.

In 3.0 the Alerting Subsystem became pluggable. Alertmanager integration is supported by the included Alertmanager Plugin. If you are using AM, the settings have to be migrated. Old settings that may remain in the configuration file will not affect the operation of Lenses. An example of migrating AM settings:

# Old layout
lenses.alert.manager.endpoints = "http://ALERTMANAGER_HOST_1:9094,http://ALERTMANAGER_HOST_2:9094,..."
lenses.alert.manager.source = "Lenses"
lenses.alert.manager.generator.url = "http://LENSES_HOST:LENSES_PORT"

# New layout
lenses.alert.plugins = [
  {
    class = "io.lenses.alerts.plugin.am.AlertManagerPlugin",
    config = {
      endpoints = "http://ALERTMANAGER_HOST_1:9094,http://ALERTMANAGER_HOST_2:9094,...",
      source = "Lenses",
      generator-url = "http://LENSES_HOST:LENSES_PORT"
    }
  }
]

In 3.0 the Alerting Subsystem became pluggable. Slack integration is supported by the included Slack Plugin. If you are using Slack, the settings have to be migrated. Old settings that may remain in the configuration file will prevent Lenses from starting up! An example of migrating Slack settings:

# Old layout
lenses.alert.plugins.slack.enabled      = true
lenses.alert.plugins.slack.webhook.url  = "https://hooks.slack.com/services/SECRET/YYYYYYYYYYY/XXXXXXXX"
lenses.alert.plugins.slack.username     = "lenses"
lenses.alert.plugins.slack.channel      = "alerts"

# New layout
lenses.alert.plugins = [
  {
    class = "io.lenses.alerts.plugin.slack.SlackAlertsPlugin",
    config = {
      webhook-url = "https://hooks.slack.com/services/SECRET/YYYYYYYYYYY/XXXXXXXX",
      username = "lenses",
      channel = "alerts"
    }
  }
]

The class of the default LDAP plugin has been renamed. If you use LDAP you must rename the class inside security.conf:

# Old
lenses.security.ldap.plugin.class = "com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin"

# New
lenses.security.ldap.plugin.class = "io.lenses.security.ldap.LdapMemberOfUserGroupPlugin"

The default value for allowing weak SSL configuration has been changed. By default in 3.0 weak SSL settings will not be allowed. This change affects mainly the Schemas and Connectors management pages:
```
# Old
lenses.allow.weak.SSL=true

# New. Default value has been changed to false
lenses.allow.weak.ssl=false
```

Client Changes¶

If you are using service accounts, now you must pass both the account name and token via the X-Kafka-Lenses-Token header:

# Old Authentication Header
X-Kafka-Lenses-Token: ServiceAccountToken

# New Authentication Header
X-Kafka-Lenses-Token: ServiceAccountName:ServiceAccountToken

There are some breaking changes in the API due to the requirements of the new, fine-grained, permissions model. If you have applications that use the API endpoints below, you must update them with the new endpoints:

Breaking API Changes Table¶
Endpoint Group	Old Endpoint	New Endpoint	Information
Kafka Topics	`GET /api/topics`	`GET /api/v1/kafka/topics`	API reference
Kafka Topic Data	`GET /api/sql/data?sql=...`	`GET /api/ws/v2/sql/execute`	API reference
SQL Validation	`GET /api/sql/validation`	n/a	Removed
Alerts	`GET /api/alerts`	`GET /api/alerts`	Response payload changed API reference
Audit	`GET /api/audit`	`GET /api/audi`	Response payload changed API reference

Helm Charts Changes¶

Zookeeper is now optional in Helm:

# Old
zookeepers:
  chroot:

# New
zookeepers:
  enabled: false
  chroot:

Security section must be adapted to the new security model of Lenses:

# Old
security:
  mode: BASIC
  kerberos:
    principal:
    keytabData: |-
    debug: false

  groups:
    - name: adminGroup
      roles:
        - admin
        - write
        - read
    - name: writeGroup
      roles:
        - read
        - write

  users:
  - username: admin
    password: admin
    displayname: LensesAdmin
    groups:
      - adminGroup

  serviceAccounts:
  - username: jenkins
    token: jenkins
    groups:
    - adminGroup

  ...

# New
security:
  # Change Default Lenses user credentials.
  # defaultUser:
  # Example
  #  username: admin
  #  password: admin
  kerberos:
    principal:
    keytabData: |-
    debug: false

  .....

Enabled Alert Plugins must be updated to the new format:

# Old
alertManagers:
enabled: false
endpoints:
  - host: alertmanager
    protocol: http
    port: 9094

sourceName: Lenses™-k8
consumersLagThreshold: 50000

slack:
enabled: false
iconUrl: https://www.lenses.io/images/logos/icon_ellipse2red.png
webhookUrl: ""
username: Lenses
channel: lenses

# New
alerts:
  enabled: true
  consumersLagThreshold: 50000
  plugins:
    - class: "io.lenses.alerts.plugin.slack.SlackAlertsPlugin"
      config: |
        {
          webhook-url="https://your-slack-webhook-url
          username=Lenses™
          channel=lenses-slack-test
        }
    - class: "io.lenses.alerts.plugin.am.AlertManagerPlugin"
      config: |
        {
          endpoints="http://your-alert-manager-url"
          source=PROD
          generator-url="http://lenses_prod:port1"
        }