Upgrade Notes

Upgrading from an older version of Lenses, requires some changes to the existing configuration files. Please familiarize with the changes described below before performing an upgrade.

For the full list of changes made for this release please check the Release Notes.

Note

Please remember that Lenses 2.2 introduced state on disk for the Data Policies module. You have to keep your Lenses Storage Directory in order for your Data Policies to transfer to 3.0. Furthermore, in 3.0 user management, audits, and alerts have been migrated to the Lenses Storage Directory, so you must make sure you persist these data in order to be able to retain them between upgrades in the future.

From 2.3 to 3.0

Application Update

Archive

Download the latest archive and extract it in a new directory. It is important to avoid extracting an archive over an older installation. You should instead remove (or rename) the old directory, then move the new into its place. Copy if needed and update your lenses.conf and security.conf files as described later in the current page (Configuration Changes). Last make sure you keep the Lenses Storage Directory intact as this is where persistent data is stored. Up to Lenses 2.3 Data Policies are stored there. Starting with 3.0, users, groups, audits and other data is stored as well.

Docker

As usual the latest tag brings the latest Lenses versions. There are also tagged releases major (3.0, 2.3) and even minor (3.0.0, 2.3.6) tags to help you better manage your upgrade path.

Starting with 3.0, the landoop/kafka-lenses-dev image can also be accessed via lensesio/box. The landoop/lenses image can also be accessed via the lensesio/lenses name. In the future the old names will be deprecated, thus it’s advised to make the switch to the new images earlier rather than later.

Docker pull the latest Lenses environment for Developers:

docker pull lensesio/box:latest

Or update you docker container for Lenses Enterprise with:

docker pull lensesio/lenses:latest

Follow the Configuration Changes section to update your settings if needed. Remember that Lenses Storage Directory should be stored in a docker volume and be kept intact between updates.

Helm

Download the latest charts and update your values.yaml as described in Configuration Changes and Helm Charts Changes sections. Remember that Lenses Storage Directory should be stored in a persistent volume and be kept intact between updates.

Cloud Installations

Use the latest version available in the marketplaces. Remember that Lenses Storage Directory should be provided as a persistent volume and be kept intact between updates.

SQL Processors

For kubernetes no action is required. New processors will use automatically the latest image. Old processors will keep working with the old image.

For Kafka Connect, please download the latest version of the connector and add it to your connect nodes whilst removing the old version. Your nodes need to be restarted for the change to take effect. A rolling restart is advised.

Configuration Changes

  • User and Group management logic became dynamic and moved into the Lenses API. You can manage users and groups via the Lenses web interface or the lenses-cli application. What this means for operators, is that the security.conf file is no longer used to setup user acounts for the BASIC authentication module, neither groups or user-group mappings.

    • If you are using the BASIC authentication module you must recreate your users via the web interface or the command line.
    • You must recreate your groups via the Web Interface or the command line. This is also a great chance to familiarize with the new, fine grained, permission system.
    • If you are using the Kerberos (SPNEGO) authentication module you must add your principals and link them to groups via the Web Interface or the command line.
    • If you are using service accounts you must recreate them via the web interface or the command line.

    Check User Management and Security Configuration to familiarize with the changes.

Note

This version comes with a default administrator user. Remember to secure your setup with a custom username and password for this user. Use SHA256 to protect the password inside security.conf.

  • In 3.0 the Alerting Subsystem became pluggable. Alertmanager integration is supported by the included Alertmanager Plugin. If you are using AM, the settings have to be migrated. Old settings that may remain in the configuration file will not affect the operation of Lenses. An example of migrating AM settings:

    # Old layout
    lenses.alert.manager.endpoints = "http://ALERTMANAGER_HOST_1:9094,http://ALERTMANAGER_HOST_2:9094,..."
    lenses.alert.manager.source = "Lenses"
    lenses.alert.manager.generator.url = "http://LENSES_HOST:LENSES_PORT"
    
    # New layout
    lenses.alert.plugins = [
      {
        class = "io.lenses.alerts.plugin.am.AlertManagerPlugin",
        config = {
          endpoints = "http://ALERTMANAGER_HOST_1:9094,http://ALERTMANAGER_HOST_2:9094,...",
          source = "Lenses",
          generator-url = "http://LENSES_HOST:LENSES_PORT"
        }
      }
    ]
    
  • In 3.0 the Alerting Subsystem became pluggable. Slack integration is supported by the included Slack Plugin. If you are using Slack, the settings have to be migrated. Old settings that may remain in the configuration file will prevent Lenses from starting up! An example of migrating Slack settings:

    # Old layout
    lenses.alert.plugins.slack.enabled      = true
    lenses.alert.plugins.slack.webhook.url  = "https://hooks.slack.com/services/SECRET/YYYYYYYYYYY/XXXXXXXX"
    lenses.alert.plugins.slack.username     = "lenses"
    lenses.alert.plugins.slack.channel      = "alerts"
    
    # New layout
    lenses.alert.plugins = [
      {
        class = "io.lenses.alerts.plugin.slack.SlackAlertsPlugin",
        config = {
          webhook-url = "https://hooks.slack.com/services/SECRET/YYYYYYYYYYY/XXXXXXXX",
          username = "lenses",
          channel = "alerts"
        }
      }
    ]
    
  • The class of the default LDAP plugin has been renamed. If you use LDAP you must rename the class inside security.conf:

    # Old
    lenses.security.ldap.plugin.class = "com.landoop.lenses.security.ldap.LdapMemberOfUserGroupPlugin"
    
    # New
    lenses.security.ldap.plugin.class = "io.lenses.security.ldap.LdapMemberOfUserGroupPlugin"
    
  • The default value for allowing weak SSL configuration has been changed. By default in 3.0 weak SSL settings will not be allowed. This change affects mainly the Schemas and Connectors management pages:

    # Old
    lenses.allow.weak.SSL=true
    
    # New. Default value has been changed to false
    lenses.allow.weak.ssl=false
    

Client Changes

  • If you are using service accounts, now you must pass both the account name and token via the X-Kafka-Lenses-Token header:

    # Old Authentication Header
    X-Kafka-Lenses-Token: ServiceAccountToken
    
    # New Authentication Header
    X-Kafka-Lenses-Token: ServiceAccountName:ServiceAccountToken
    
  • There are some breaking changes in the API due to the requirements of the new, fine-grained, permissions model. If you have applications that use the API endpoints below, you must update them with the new endpoints:

    Breaking API Changes Table
    Endpoint Group Old Endpoint New Endpoint Information
    Kafka Topics GET /api/topics GET /api/v1/kafka/topics API reference
    Kafka Topic Data GET /api/sql/data?sql=... GET /api/ws/v2/sql/execute API reference
    SQL Validation GET /api/sql/validation n/a Removed
    Alerts GET /api/alerts GET /api/alerts
    Response payload changed
    Audit GET /api/audit GET /api/audi
    Response payload changed

Helm Charts Changes

  • Zookeeper is now optional in Helm:

    # Old
    zookeepers:
      chroot:
    
    # New
    zookeepers:
      enabled: false
      chroot:
    
  • Security section must be adapted to the new security model of Lenses:

    # Old
    security:
      mode: BASIC
      kerberos:
        principal:
        keytabData: |-
        debug: false
    
      groups:
        - name: adminGroup
          roles:
            - admin
            - write
            - read
        - name: writeGroup
          roles:
            - read
            - write
    
      users:
      - username: admin
        password: admin
        displayname: LensesAdmin
        groups:
          - adminGroup
    
      serviceAccounts:
      - username: jenkins
        token: jenkins
        groups:
        - adminGroup
    
      ...
    
    # New
    security:
      # Change Default Lenses user credentials.
      # defaultUser:
      # Example
      #  username: admin
      #  password: admin
      kerberos:
        principal:
        keytabData: |-
        debug: false
    
      .....
    
  • Enabled Alert Plugins must be updated to the new format:

    # Old
    alertManagers:
    enabled: false
    endpoints:
      - host: alertmanager
        protocol: http
        port: 9094
    
    sourceName: Lenses™-k8
    consumersLagThreshold: 50000
    
    slack:
    enabled: false
    iconUrl: https://www.lenses.io/images/logos/icon_ellipse2red.png
    webhookUrl: ""
    username: Lenses
    channel: lenses
    
    # New
    alerts:
      enabled: true
      consumersLagThreshold: 50000
      plugins:
        - class: "io.lenses.alerts.plugin.slack.SlackAlertsPlugin"
          config: |
            {
              webhook-url="https://your-slack-webhook-url
              username=Lenses™
              channel=lenses-slack-test
            }
        - class: "io.lenses.alerts.plugin.am.AlertManagerPlugin"
          config: |
            {
              endpoints="http://your-alert-manager-url"
              source=PROD
              generator-url="http://lenses_prod:port1"
            }