# Steps for SideBySide Upgrade

## Prerequisites

{% stepper %}
{% step %}
**Have HQ deployed**

The first step is to have HQ deployed and *agent\_key* being obtained upon [environment creation](https://docs.lenses.io/latest/deployment/upgrade-to-lenses-6/broken-reference).
{% endstep %}

{% step %}
**Prepare Postgres Database**

{% hint style="warning" %}
**H2 is not recommended for production environments.**
{% endhint %}

For any other purposes, it is highly recommended to use a PostgreSQL (preferred) or Microsoft SQL Server database. **Multiple agents can use the same Postgres database, but in that case, you must ensure that each Agent uses a different schema.**

Therefore, in preparatio,n you must ensure:

* [x] Postgres instance
* [x] Postgres schema per Agent
* [x] Postgres user and password per Agent

For non-production environments, you can rely on the embedded H2 database.
{% endstep %}

{% step %}
**Convert all connections / alerts / channels to Provisioning**

There were multiple ways which Lenses resources could have been managed in the past:

1. Wizard
2. Provisioning v1
3. Provisioning v2

For Lenses (6) Agent, it is recommended that all connections (*kafka, schema-registry, connect,*...) are kept inside of *provisioning.yaml* file in *version 2*. [automation](https://docs.lenses.io/latest/deployment/configuration/agent/automation "mention")

Differences in provisioning between version 1 and 2 can be seen below:

{% tabs %}
{% tab title="Provisioning version 1" %}
{% code title="provisioning.yaml" %}

```yaml
license:
  fileRef:
    inline: '{"source":"Landoop LTD","clientId":"Lenses Dev","details":"kafka-lenses","key":"eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0.AqO6Ax-o-4T0WKFX7eCGFRu329wxplkZuWrGdhyncrhBfh9higjsZA.-uGCXWjULTzb7-3ROfsRhw.olmx6FR7FH7c2adHol0ipokHF6jOo6LTDtoFOSPWfqKxbA3yI-CUqlyo_-Obin7MSA4KqXBLpXOvP72EJhIYuyqkxUVGRoXHF0Oj2V6kzdDcmjJHbMB4VTxdE8YBAYbPXzEXdhq7lZy4fxHHhYxAsRATCtqf7t7TQCE0TWOiSHvLwyD7xMK2X47KiKbnNlNvqeVnnjLUMMd7vzA5dTft48wJm2D5HJNZ0mS32gTaiiExT5nqolToL0KYIOpRiT00MTQkGlBdagVigc-DZBPM0ZTP5wuLkwdk4XbfoQKaWC4qaYA6VpGgQg03Mo1W4ljlqRy0N4cPQ-l4Mi1XV9VK-825-zhyxzPrxef5Zct2nzVEJ9MbWy0-xuf6THX4q2X8zmz_KiHoA-hBWjebv_2R9479ldGj0h-vm9htVD59_6RBOGb0rT4XSS-4_CGYBZzv5PIPpLdnVbkr_qjsxCI0BO7tPKoyxXg2qh4YQbn3wn5MqsE9yR2BbRaso9MSPFlF8PxqR7A4qrKJjn_mPlcrR-XGf0ua2XfWCVe4ngcWpzssYHcJJD80APyZgzneIw2dSaO0enfFYUq6avqGSeoG7VC9zYACfUdofdlULH2azmptJ2Jzw3ggpLR7ZzZ9QrySXTUB2jkzrqiHyM9fqIXUVwAkAJMcBuwF5zY5B_ChA69Uj_-s-S1RITBbg5wtB3LuHyJGtTo4fuYY75F_OL9Cwp7gcpa5u0M_wWZlx70j_6jCb-ogvghALbHY8OPeWz_1-3bvJM9T_jmjKy0FDt6x8FJV1lgMMR0j1RiUeauUMsnd4TNUYAH50mFwtK5PU-Iq.U4LwNfOL2JB4vzBMvo3Hig"}'
connections:
  kafka:
    tags: []
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
        - PLAINTEXT:///localhost:9092
      protocol: PLAINTEXT
      # metrics
      ## JMX
      metricsPort: 9581
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19581
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  zookeeper:
    tags: []
    templateName: Zookeeper
    configurationObject:
      zookeeperUrls:
        - localhost:2181
      zookeeperSessionTimeout: 10000
      zookeeperConnectionTimeout: 10000
      # metrics
      ## JMX
      metricsPort: 9585
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19585
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  schema-registry:
    templateName: SchemaRegistry
    tags: [ ]
    configurationObject:
      schemaRegistryUrls:
        - http://localhost:8081
      additionalProperties: { }
      # metrics
      ## JMX
      metricsPort: 9582
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19582
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  connect-cluster-dev-1:
    templateName: KafkaConnect
    tags: []
    configurationObject:
      workers:
        - http://localhost:8083
      aes256Key: PasswordPasswordPasswordPassword
      # metrics
      ## JMX
      metricsPort: 9584
      metricsType: JMX
      metricsSsl: false
      ## JOLOKIA
      # metricsPort: 19584
      # metricsType: JOLOKIAG # or JOLOKIAP
      # metricsSsl: false
      # metricsHttpSuffix: "/jolokia/"

  my-prometheus: {"configuration":[{"key":"endpoints","value":["https://am.acme.com"]}],"tags":["prometheus","monitoring","metrics"],"templateName":"PrometheusAlertmanager"}

```

{% endcode %}
{% endtab %}

{% tab title="Provisioning version 2" %}
{% code title="provisioning.yaml" %}

```yaml
lensesHq:
  - configuration:
      agentKey:
        value: ${LENSESHQ_AGENT_KEY}
      port:
        value: 10000
      server:
        value: lenses-hq
    name: lenses-hq
    tags: ['hq']
    version: 1
kafka:
  - name: kafka
    version: 1
    tags: [ 'kafka', 'dev' ]
    configuration:
      metricsType:
        value: JMX
      metricsPort:
        value: 9581
      kafkaBootstrapServers:
        value: [PLAINTEXT://demo-kafka:19092]
      protocol:
        value: PLAINTEXT
confluentSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: [ 'dev' ]
    configuration:
      schemaRegistryUrls:
        value: [http://demo-kafka:8081]
      metricsType:
        value: JMX
      metricsPort:
        value: 9582
connect:
  - name: dev
    version: 1
    tags: [ 'dev' ]
    configuration:
      workers:
        value: [http://demo-kafka:8083]
      aes256Key:
        value: 0123456789abcdef0123456789abcdef
      metricsType:
        value: JMX
      metricsPort:
        value: 9584%
```

{% endcode %}
{% endtab %}
{% endtabs %}

More about other configuration options in *provisioning.yaml ->* [automation](https://docs.lenses.io/latest/deployment/configuration/agent/automation "mention")

If you are curious about how to properly create the *provisioning.yaml* file, you can read more on [how-to-convert-wizard-mode-to-provisioning-mode](https://docs.lenses.io/latest/deployment/upgrade-to-lenses-6/how-to-convert-wizard-mode-to-provisioning-mode "mention")
{% endstep %}
{% endstepper %}

## Steps to upgrade to Agent

{% hint style="warning" %}
In case you have to migrate SQL Processors and Postgres is not set as a data store, please read [#optional-migrate-of-sql-processors-and-the-rest](#optional-migrate-of-sql-processors-and-the-rest "mention") first.
{% endhint %}

{% stepper %}
{% step %}
**Follow the prerequisite steps above**

Steps:

* [x] Deploy Lenses HQ
* [x] Prepare Database
* [x] Prepare Provision files

Once you've completed the steps above, you'll be ready to proceed and deploy the Agent.
{% endstep %}

{% step %}
**Pick a deployment method**

There are multiple deployment methods for the Agent deployment. Please choose one from [installation](https://docs.lenses.io/latest/deployment/installation "mention")

{% hint style="success" %}
It is **perfectly safe to run your older installation of Lenses 4.x/5.x and at the side Agent that is connecting to the existing Kafka Cluster.**

Lenses ⅘ and Agent behaves as just another KafkaAdminClient that is connecting to your Kafka cluster and therefore they can live next to each other.
{% endhint %}

{% hint style="danger" %}
**Two** **Lenses** instances **shouldn't be** **connecting** to the **same** **database.**
{% endhint %}
{% endstep %}

{% step %}
**Migrate of Lenses 5.5 groups and data policies**

{% hint style="success" %}
Migrating of :

* data policies
* alerts
* sql processors

i**s not necessary in case Agent will re-use the same database as Lenses 5.**
{% endhint %}

Once you confirm that:

* [x] HQ works and users can login via SAML/SSO
* [x] Agent is connected to HQ and you (admin) can inspect it

It is time to move towards migrating Lenses ⅘ groups to:

* HQ Groups / Roles / Permissions;
* Data policies

In order to do so, we have tooling that can help you [Lenses Migration Tool](https://github.com/lensesio-ps/lenses-migration-tool).

{% hint style="warning" %}
Be aware, cloning of **alerts** is not available yet via script above.
{% endhint %}
{% endstep %}

{% step %}
**(Optional) Migrate of SQL Processors and the rest**

In case new database will be used for Agent upgrade path it it highly recommended to pick <a href="steps-for-in-place-upgrade" class="button primary">In Place upgrade</a>.

SQL Processors are stored within Lenses ⅘ database. Same will continue to work even if Lenses instance is not running anymore, but in order to preserve its configuration and possibility for Agent to be aware of their whereabouts following step has to be done.

**Old Postgres to new Postgres database**

This requires multiple steps:

1. Stop Lenses 5 instance;
2. Backup Postgres database Lenses 5 was using;
3. Load the same database to new Postgres database;
4. Start Lenses Agent (v6).

**Old H2 database to new H2 database**

{% hint style="info" %}
Under construction :construction:
{% endhint %}

**Old H2 database to new Postgres database**

{% hint style="warning" %}
Only available with if upgrade to Lenses Agent v6 is being done from Lenses 5.5 and onwards
{% endhint %}

Process of migration looks a follows:

1. **Create a database and user in PostgreSQL for Lenses ⅘ to use.**\\

   ```bash
   # login as superuser and add Lenses role and database
   psql -U postgres -d postgres <<EOF
   CREATE ROLE lenses WITH LOGIN PASSWORD 'changeme';
   CREATE DATABASE lenses OWNER lenses;
   EOF
   ```
2. **Make a backup**\
   Take a backup of your current embedded database (its path is controlled via the setting `lenses.storage.directory`). Just copy the directory.
3. **Prepare Lenses conf**\
   Edit the Lenses configuration file (`lenses.conf`), adding the PostgreSQL settings. E.g:\\

   ```apacheconf
   lenses.storage.postgres.password="changeme"
   lenses.storage.postgres.host="my-postgres-server"
   lenses.storage.postgres.port=5431 # optional, defaults to 5432
   lenses.storage.postgres.username="lenses"
   lenses.storage.postgres.database="lenses"
   ```
4. Restart Lenses. It will perform the migration automatically.
5. Once everything looks good, delete the directory containing the embedded database, and remove the key `lenses.storage.directory` from your `lenses-agent.conf`.
6. Perform Lenses Agent upgrade [#steps-to-upgrade-to-agent](#steps-to-upgrade-to-agent "mention")
   {% endstep %}
   {% endstepper %}