Steps for SideBySide Upgrade

Guide on how to migrate your Lenses 5 instance to Lenses 6 Agent

Prerequisites

1

Have HQ deployed

The first step is to have HQ deployed and agent_key being obtained upon environment creation.

2

Prepare Postgres Database

H2 is not recommended for production environments.

For any other purposes, it is highly recommended to use a PostgreSQL (preferred) or Microsoft SQL Server database. Multiple agents can use the same Postgres database, but in that case, you must ensure that each Agent uses a different schema.

Therefore, in preparatio,n you must ensure:

• Postgres instance

• Postgres schema per Agent

• Postgres user and password per Agent

For non-production environments, you can rely on the embedded H2 database.

3

Convert all connections / alerts / channels to Provisioning

There were multiple ways which Lenses resources could have been managed in the past:

1. Wizard

2. Provisioning v1

3. Provisioning v2

For Lenses (6) Agent, it is recommended that all connections (kafka, schema-registry, connect,...) are kept inside of provisioning.yaml file in version 2. Provisioning

Differences in provisioning between version 1 and 2 can be seen below:

Provisioning version 1

provisioning.yaml

license:
  fileRef:
    inline: '{"source":"Landoop LTD","clientId":"Lenses Dev","details":"kafka-lenses","key":"eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.AqO6Ax-o-4T0WKFX7eCGFRu329wxplkZuWrGdhyncrhBfh9higjsZA.-uGCXWjULTzb7-3ROfsRhw.olmx6FR7FH7c2adHol0ipokHF6jOo6LTDtoFOSPWfqKxbA3yI-CUqlyo_-Obin7MSA4KqXBLpXOvP72EJhIYuyqkxUVGRoXHF0Oj2V6kzdDcmjJHbMB4VTxdE8YBAYbPXzEXdhq7lZy4fxHHhYxAsRATCtqf7t7TQCE0TWOiSHvLwyD7xMK2X47KiKbnNlNvqeVnnjLUMMd7vzA5dTft48wJm2D5HJNZ0mS32gTaiiExT5nqolToL0KYIOpRiT00MTQkGlBdagVigc-DZBPM0ZTP5wuLkwdk4XbfoQKaWC4qaYA6VpGgQg03Mo1W4ljlqRy0N4cPQ-l4Mi1XV9VK-825-zhyxzPrxef5Zct2nzVEJ9MbWy0-xuf6THX4q2X8zmz_KiHoA-hBWjebv_2R9479ldGj0h-vm9htVD59_6RBOGb0rT4XSS-4_CGYBZzv5PIPpLdnVbkr_qjsxCI0BO7tPKoyxXg2qh4YQbn3wn5MqsE9yR2BbRaso9MSPFlF8PxqR7A4qrKJjn_mPlcrR-XGf0ua2XfWCVe4ngcWpzssYHcJJD80APyZgzneIw2dSaO0enfFYUq6avqGSeoG7VC9zYACfUdofdlULH2azmptJ2Jzw3ggpLR7ZzZ9QrySXTUB2jkzrqiHyM9fqIXUVwAkAJMcBuwF5zY5B_ChA69Uj_-s-S1RITBbg5wtB3LuHyJGtTo4fuYY75F_OL9Cwp7gcpa5u0M_wWZlx70j_6jCb-ogvghALbHY8OPeWz_1-3bvJM9T_jmjKy0FDt6x8FJV1lgMMR0j1RiUeauUMsnd4TNUYAH50mFwtK5PU-Iq.U4LwNfOL2JB4vzBMvo3Hig"}'
connections:
  kafka:
    tags: []
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
        - PLAINTEXT:///localhost:9092
      protocol: PLAINTEXT
      # metrics
      ## JMX
      metricsPort: 9581
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19581
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  zookeeper:
    tags: []
    templateName: Zookeeper
    configurationObject:
      zookeeperUrls:
        - localhost:2181
      zookeeperSessionTimeout: 10000
      zookeeperConnectionTimeout: 10000
      # metrics
      ## JMX
      metricsPort: 9585
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19585
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  schema-registry:
    templateName: SchemaRegistry
    tags: [ ]
    configurationObject:
      schemaRegistryUrls:
        - http://localhost:8081
      additionalProperties: { }
      # metrics
      ## JMX
      metricsPort: 9582
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19582
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  connect-cluster-dev-1:
    templateName: KafkaConnect
    tags: []
    configurationObject:
      workers:
        - http://localhost:8083
      aes256Key: PasswordPasswordPasswordPassword
      # metrics
      ## JMX
      metricsPort: 9584
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19584
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  my-prometheus: {"configuration":[{"key":"endpoints","value":["https://am.acme.com"]}],"tags":["prometheus","monitoring","metrics"],"templateName":"PrometheusAlertmanager"}

Provisioning version 2

More about other configuration options in provisioning.yaml -> Provisioning

If you are curious about how to properly create the provisioning.yaml file, you can read more on How to convert Wizard Mode to Provisioning Mode

Steps to upgrade to Agent

In case you have to migrate SQL Processors and Postgres is not set as a data store, please read (Optional) Migrate of SQL Processors and the rest first.

1

Follow the prerequisite steps above

Steps:

• Deploy Lenses HQ

• Prepare Database

• Prepare Provision files

Once you've completed the steps above, you'll be ready to proceed and deploy the Agent.

2

Pick a deployment method

There are multiple deployment methods for the Agent deployment. Please choose one from Installation

It is perfectly safe to run your older installation of Lenses 4.x/5.x and at the side Agent that is connecting to the existing Kafka Cluster.

Lenses ⅘ and Agent behaves as just another KafkaAdminClient that is connecting to your Kafka cluster and therefore they can live next to each other.

Two Lenses instances shouldn't be connecting to the same database.

3

Migrate of Lenses 5.5 groups and data policies

Migrating of :

• data policies

• alerts

• sql processors

is not necessary in case Agent will re-use the same database as Lenses 5.

Once you confirm that:

• HQ works and users can login via SAML/SSO

• Agent is connected to HQ and you (admin) can inspect it

It is time to move towards migrating Lenses ⅘ groups to:

• HQ Groups / Roles / Permissions;

• Data policies

In order to do so, we have tooling that can help you Lenses Migration Tool.

Be aware, cloning of alerts is not available yet via script above.

4

(Optional) Migrate of SQL Processors and the rest

In case new database will be used for Agent upgrade path it it highly recommended to pick In Place upgrade.

SQL Processors are stored within Lenses ⅘ database. Same will continue to work even if Lenses instance is not running anymore, but in order to preserve its configuration and possibility for Agent to be aware of their whereabouts following step has to be done.

Old Postgres to new Postgres database

This requires multiple steps:

1. Stop Lenses 5 instance;

2. Backup Postgres database Lenses 5 was using;

3. Load the same database to new Postgres database;

4. Start Lenses Agent (v6).

Old H2 database to new H2 database

Under construction 🚧

Old H2 database to new Postgres database

Only available with if upgrade to Lenses Agent v6 is being done from Lenses 5.5 and onwards

Process of migration looks a follows:

1. Create a database and user in PostgreSQL for Lenses ⅘ to use.\

   # login as superuser and add Lenses role and database
   psql -U postgres -d postgres <<EOF
   CREATE ROLE lenses WITH LOGIN PASSWORD 'changeme';
   CREATE DATABASE lenses OWNER lenses;
   EOF

2. Make a backup Take a backup of your current embedded database (its path is controlled via the setting lenses.storage.directory). Just copy the directory.

3. Prepare Lenses conf Edit the Lenses configuration file (lenses.conf), adding the PostgreSQL settings. E.g:\

   lenses.storage.postgres.password="changeme"
   lenses.storage.postgres.host="my-postgres-server"
   lenses.storage.postgres.port=5431 # optional, defaults to 5432
   lenses.storage.postgres.username="lenses"
   lenses.storage.postgres.database="lenses"

4. Restart Lenses. It will perform the migration automatically.

5. Once everything looks good, delete the directory containing the embedded database, and remove the key lenses.storage.directory from your lenses-agent.conf.

6. Perform Lenses Agent upgrade Steps to upgrade to Agent