6.0 Preview

New

Lenses 6.0 introduces a new service, called HQ, acting as portal for multi-kafka environments.

• New HQ service

• IAM (Identity & Access Management). This has moved from each Lenses instant to a global location in the new HQ service

• Global SQL Studio

• Global Data Catalogue

• Community License: You can no use Lenses without a license (community license key still rebuild and bundled in docker-compose) or expiry but the following restrictions apply:

  • No SSO

  • Maximum of two environments (Kafka clusters) can be connected

  • Two Users with one an admin user

  • Two Service Accounts

  • Two Groups

  • Two Roles

  • No Backup / Restore for topics to S3

Breaking

1. H2 embedded database is no longer support.

2. Lenses 5.x permission model is replaced by global IAM. You must recreate the roles and groups in HQ

3. Connection management in the agent is via file Provisioning only.

Run the following command:

wget https://lenses.io/preview -O docker-compose.yml
docker compose pull
ACCEPT_EULA=true docker compose up

Once the images are pulled and containers started, you can login here with admin/admin and explore.

The quick start uses a docker compose file to:

Packages

• We have made new alpha release 16:

  • Agent image:
  • HQ image:
• New Helm version 16 for agent and for the HQ:

HQ Changelog

Agent

Changes in provisioning connection to HQ

In the provisioning, there has been slight adjustment in connection naming with HQ.

Changes:

• grpcServer has been renamed to lensesHq

• apiKey has been renamed to agentKey

Known issues

With the new version of Agent, HQ connection in provisioning has changed which requires complete recreation of database. Following log message will indicate it:

The quick start is for local development, with a local Kafka. This guide takes you through manually deploying HQ and an Agent to connect to your Kafka clusters.

For more detailed guides on the Helm, Docker and Linux see .

How to connect to your Kafka?

To deploy Lenses against your environments you need to:

Prerequisites

EULA acceptance

To start HQ and an Agent you have to accept the .

For HQ, in the config.yaml set:

Kafka

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud. Supported providers include:

1. Confluent Platform & Cloud

2. AWS MSK & AWS MSK Serverless

3. Aiven

4. IBM Event Streams

5. Azure HDInsight & EventHubs

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

Postgres

HQ and Agents can share the same instance, by either using a separate database or schema for HQ and each agent, depending on your networking needs.

1. Postgres server running version 9.6 or higher.

Database Role

The recommended configuration is to create a dedicated login role and database for the HQ and each Agent, setting the HQ or Agent role as the database or schema owner. Both the agent and HQ need credentials, create a role for each.

Networking

2. JMX connectivity - Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and Open Metrics (MSK).

Kafka ACLs

The agent requires access to your Kafka cluster. If ACLs are enable you will need to allow the Agent access.

SSO (optional)

How Lenses Works

Lenses has two components:

HQ Changelog

HQ configuration file TOML -> YAML

In the past, HQ has been using TOML file format. As we want to reduce differences in file formats between Agent and HQ as much as possible, this was the first step.

Postgres changes

• Postgres connection URI is not being built within config.yaml but in backend runtime;

  • parameter group has changed from postgres to storage.postgres.*

• In the previous version, schema was defined as a part of extraParamSpecs. In the new version schema is now defined as a separate property storage.postgres.database.schema;

• Property extraParamSpecs is renamed to params;

Parameter group api changes

Parameter group api has been renamed to http and following parameters are not part of it anymore:

• administrators;

• saml;

New parameter auth

Property auth is being derived from property api (now. http).

Parameters that has been moved from http to auth are following:

• administrators;

• saml;

Aurora support

HQ has been tested against Aurora (Postgres) and is compatible.

Pod restart on configmap checksum change

In case of any changes in ConfigMap and after executing helm upgrade HQ pod will be automatically restarted as well therefore no need for manual interventions.

TL; DR Changes in values.yaml

lensesHq:
  auth:
    administrators:
      - admin@example.com
      - admin2@example.com
    saml:
      baseURL: "https://"
      entityID: "https://"
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: hq-tls-mock-saml-metadata
        secretKeyName: metadata.xml
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://saml.example.com/entityid" validUntil="2034-09-23T08:10:34.764Z">
            <md:IDPSSODescriptor WantAuthnRequestsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
              <md:KeyDescriptor use="signing">
                <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                  <ds:X509Data>
                    <ds:X509Certificate>MIIC4jCCAcoCCQC33wnybT5QZDANBgkqhkiG9w0BAQsFADAyMQswCQYDVQQGEwJV
          ...
          m0eo2USlSRTVl7QHRTuiuSThHpLKQQ==</ds:X509Certificate>
                  </ds:X509Data>
                </ds:KeyInfo>
              </md:KeyDescriptor>
              <md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mocksaml.com/api/saml/sso"/>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mocksaml.com/api/saml/sso"/>
            </md:IDPSSODescriptor>
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "manual"
    tls:
      enabled: false
      cert:
        referenceFromSecret: false
        stringData: |
          -----BEGIN CERTIFICATE-----
          MIIECDCCAvCgAwIBAgIRAIiph6RQMbGCABOaJO94PbowDQYJKoZIhvcNAQELBQAw
          ...
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  agents:
    address: ":10000"
    tls:
      enabled: false
      verboseLogs: false
      cert:
        referenceFromSecret: false
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
        stringData: |
          -----BEGIN CERTIFICATE-----
          MIIECDCCAvCgAwIBAgIRAIiph6RQMbGCABOaJO94PbowDQYJKoZIhvcNAQELBQAw
          gYUxFTATBgNVBAYTDEFua2gtTW9ycG9yazEaMBgGA1UEChMRVW5zZWVuIFVuaXZl
          .....
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - https://
      - http://
  storage:
    postgres:
      #host: postgres-postgresql.postgres.svc.cluster.local
      host: 
      port: 5432
      #username: lenses
      username: postgres
      database: hq
      passwordSecret:
        type: "externalSecret"
        name: postgres-aurora
        key:  password
        externalSecret:
          secretStoreRef:
            clusterSecretStore:
              name: enjoy3-secrets
  logger:
    mode: "text"
    level: "debug"

  # Additional env variables appended to deployment
  # Follows the format of [EnvVar spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#envvar-v1-core)
  additionalEnv:
    # - name: FOO
    #   value: bar

  # Disable livenessProbe, used while debugging
  livenessProbe:
    enabled: false

  # Pause execution of Lenses start up script to allow the user to login into the container and
  # check the running environment, used while debugging
  pauseExec:
    enabled: false

  monitoring:
    port: 9090

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: false
  api:
    address: ":8080"
    accessControlAllowOrigin: '["http://localhost:8080"]'
    administrators: '["admin@example.com"]'
    saml:
      baseUrl: "http://localhost:8080"
      entityId: "http://localhost:8080"
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://saml.example.com/entityid" validUntil="2034-09-23T08:10:34.764Z">
            <md:IDPSSODescriptor WantAuthnRequestsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
              <md:KeyDescriptor use="signing">
                <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                  <ds:X509Data>
                    <ds:X509Certificate>MIIC4jCCAcoCCQC33wnybT5QZDANBgkqhkiG9w0BAQsFADAyMQswCQYDVQQGEwJV
          SzEPMA0GA1UECgwGQm94eUhRMRIwEAYDVQQDDAlNb2NrIFNBTUwwIBcNMjIwMjI4
          MjE0NjM4WhgPMzAyMTA3MDEyMTQ2MzhaMDIxCzAJBgNVBAYTAlVLMQ8wDQYDVQQK
          DAZCb3h5SFExEjAQBgNVBAMMCU1vY2sgU0FNTDCCASIwDQYJKoZIhvcNAQEBBQAD
          ggEPADCCAQoCggEBALGfYettMsct1T6tVUwTudNJH5Pnb9GGnkXi9Zw/e6x45DD0
          RuRONbFlJ2T4RjAE/uG+AjXxXQ8o2SZfb9+GgmCHuTJFNgHoZ1nFVXCmb/Hg8Hpd
          4vOAGXndixaReOiq3EH5XvpMjMkJ3+8+9VYMzMZOjkgQtAqO36eAFFfNKX7dTj3V
          pwLkvz6/KFCq8OAwY+AUi4eZm5J57D31GzjHwfjH9WTeX0MyndmnNB1qV75qQR3b
          2/W5sGHRv+9AarggJkF+ptUkXoLtVA51wcfYm6hILptpde5FQC8RWY1YrswBWAEZ
          NfyrR4JeSweElNHg4NVOs4TwGjOPwWGqzTfgTlECAwEAATANBgkqhkiG9w0BAQsF
          AAOCAQEAAYRlYflSXAWoZpFfwNiCQVE5d9zZ0DPzNdWhAybXcTyMf0z5mDf6FWBW
          5Gyoi9u3EMEDnzLcJNkwJAAc39Apa4I2/tml+Jy29dk8bTyX6m93ngmCgdLh5Za4
          khuU3AM3L63g7VexCuO7kwkjh/+LqdcIXsVGO6XDfu2QOs1Xpe9zIzLpwm/RNYeX
          UjbSj5ce/jekpAw7qyVVL4xOyh8AtUW1ek3wIw1MJvEgEPt0d16oshWJpoS1OT8L
          r/22SvYEo3EmSGdTVGgk3x3s+A0qWAqTcyjr7Q4s/GKYRFfomGwz0TZ4Iw1ZN99M
          m0eo2USlSRTVl7QHRTuiuSThHpLKQQ==</ds:X509Certificate>
                  </ds:X509Data>
                </ds:KeyInfo>
              </md:KeyDescriptor>
              <md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mocksaml.com/api/saml/sso"/>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mocksaml.com/api/saml/sso"/>
            </md:IDPSSODescriptor>
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
    tls:
      enabled: false
  # Find more details in https://docs.lenses.io/current/installation/kubernetes/helm/#helm-storage
  ## Postgres template example: "postgres://[username]:[pwd]@[host]:[port]/[database]?sslmode=require"
  postgres:
    host: postgres-postgresql.postgres.svc.cluster.local
    port: 5432
    username: lenses
    database: hq
    #extraParamSpecs: "?sslmode=disable"
    passwordSecret:
      type: "externalSecret"
      name: postgres
      key:  password
      externalSecret:
        secretStoreRef:
          clusterSecretStore:
            name: [CLUSTER_SECRET_STORE_NAME]

  logger:
    mode: "text"
    level: "debug"

  # Additional env variables appended to deployment
  # Follows the format of [EnvVar spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#envvar-v1-core)
  additionalEnv:
    # - name: FOO
    #   value: bar

  # Disable livenessProbe, used while debugging
  livenessProbe:
    enabled: true

  monitoring:
    port: 9090

Agent Changelog

Agent Key reference change

Previously environment variable known as LENSES_HQ_AGENT_KEY that was referenced in provisioning.yaml and stores the agentKey value has been renamed to LENSESHQ_AGENT_KEY.

Known issues

Since newest version of Lenses HQ and Agent bring breaking changes following issues can happen.

Database migration

Upon doing helm upgrade HQ can fail with following error log:

Fatal: Init db error" err="migrate database: database has been migrated beyond what we understand: the current is 17; the target is 2" version=v6.0.0-alpha.14

In order to fix it, following command has to be run on the postgres database:

DELETE FROM goose_db_version WHERE id > 2;

In case SQL command cannot be run, database has to be cleared as if one is starting from scratch.

Configure HQ

HQ is configured via by one file, config.yaml. The docker compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

 hq.config.yaml:
    content: |
      # ACCEPT THE LENSES EULA
      license:
        acceptEULA: true
      database:
        host: postgres:5432
        username: [YOUR_POSTGRES_YOUR_NAME]
        password: lenses
        database: hq

Authentication

Currently HQ supports:

1. Basic Authentication (default)

2. SAML

For this example we will use basic authentication, for information on configuring other methods, see Authentication and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

docker-compose up hq

You can now log in your browser with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

1. Login into HQ and create an environment, Environments->New Environment.

2. At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

➜  out ./hq-linux-amd64 environments
Manage Environments.

Usage:
  hq environments [command]

Aliases:
  environments, e, env, envs

Available Commands:
  create      Creates a new environment.
  delete      Deletes an environment.
  get         Retrieves a single environment by name.
  list        Lists all environments
  metadata    Manages environment metadata.
  update      Updates an environment.
  watch       Watch live environment updates.

Configure the Agent

The Agent is configured via two files:

• lenses.conf - holds low-level configuration options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file

• provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

 agent.lenses.conf:
    content: |
      lenses.storage.postgres.host=[YOUR_POSTGRES_INSTANCE]
      lenses.storage.postgres.port=[YOUR_POSTGRES_PORT]
      lenses.storage.postgres.database=agent
      lenses.storage.postgres.username=lenses
      lenses.storage.postgres.password=lenses

Adding a HQ Connection

The Agent Key for an environment needs to be added to the agent.provisioning.yaml key in the docker compose file.

agent.provisioning.yaml:
    content: |
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace ${{LENSESHQ_AGENT_KEY}} with the Agent Key for the environment that you want to link to.

For more information on the configuration of the connection to HQ see here.

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this update the agent.provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

1. Add the following for a basic plaintext connection to a Kafka broker, if you are using a different authentication mechanism adjust accordingly.

2. Remove, or adjust the Kafka (kafka-demo), Schema Registry and Connect services in the default docker-compose file.

agent.provisioning.yaml:
    content: |
      kafka:
        - name: kafka
          version: 1
          tags: [ 'kafka', 'dev' ]
          configuration:
            metricsType:
              value: JMX
            metricsPort:
              value: 9581
            kafkaBootstrapServers:
              value: PLAINTEXT://[YOUR_BOOTSTRAP_BROKER:PORT] 
            protocol:
              value: PLAINTEXT
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace [YOUR_BOOTSTRAP_BROKER:PORT] with the bootstrap brokers and ports for the Kafka cluster you want the Agent to connect to.

Start the Agent

To start Agent, run the following docker command:

docker-compose up agent

Once the agent fully starts, it will report as connected in HQ, allowing you to explore your Kafka environments.

Prerequisites

• Kubernetes 1.23+

• Helm 3.8.0+

• Running Postgres instance:

  • database for HQ;

  • username (and password) that has access to HQ database;

• Optional External secret operator (in case of ExternalSecret usage)

Configure HQ

In order to configure properly HQ we have to understand parameter groups that the Chart offers.

Under the lensesHq parameter there some key parameter groups that are used to setup HQ:

1. storage

   • definition of connection towards database (Postgres is the only storage option)

2. auth

   • Password based authentication configuration

   • SAML / SSO configuration

   • definition of administrators or first users to access the HQ

3. http

   • defines port under which HQ will be available for end users

   • defines values of special headers and cookies

   • types of connection such as TLS and non-TLS definitions

4. agents

   • defines connection between HQ and the Agent such as port where HQ will be listening for agent connections.

   • types of connection such as TLS and non-TLS definitions

5. license

6. monitoring

   • controls the metrics settings where Prometheus alike metrics will be exposed

7. loggers

   • definition of logging level for HQ

Moving forward, in the same order you can start configuring your Helm chart.

lensesHq:
  storage:
    postgres:
      enabled: true
      host: ""
      port: 
      username: ""
      database: ""
      schema: ""
      tls: 
      params: {}
      passwordSecret:
        type: ""

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses
      useSecretForUsername:
        enabled: true
        existingSecret:
          name: my-secret
          key: username

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.playground.svc.cluster.local
      port: 5432
      username: lenses
      database: lenseshq
      passwordSecret:
        type: "externalSecret"
        # Secret name where database password will be read from
        name: hq-password
        # Key name under secret where database password is stored
        key: password
        externalSecret:
          secretStoreRef:
            clusterSecretStore:
              name: enjoy2-secrets            

 lensesHq:
   storage:  
     postgres:
       enabled: true
       host: postgres-postgresql.playground.svc.cluster.local
       port: 5432
       username: lenses
       database: lenseshq
       passwordSecret:
         type: "precreated"
         # Secret name where database password will be read from
         name: hq-password
         # Key from secret's data where database password is being stored
         key: postgres-password

 lensesHq:
   storage:
     postgres:
       enabled: true
       host: [POSTGRES_HOSTNAME]
       port: 5432
       username: lenses
       database: lenseshq
       passwordSecret:
         type: "createNew"
         # name of a secret that will be created
         name: [K8s_SECRET_NAME]
         # Database password
         password: [DATABASE_USER_PASSWORD]

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses
      params:
        sslmode: require

lensesHq:
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG
    administrators:
      - admin
      - admin@example.com
      - admin2@example.com
    saml:
      enabled: true
      baseURL: ""
      entityID: ""
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: ""
        secretKeyName: ""
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          </md:EntityDescriptor>
  
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
      uiRootURL: "/"
      groupAttributeKey: "groups"

lensesHq:
  auth:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    administrators:
      - admin@example.com
      - admin2@example.com
    saml:
      baseURL: ""
      entityID: ""
      metadata:
        referenceFromSecret: true
        secretName: hq-tls-mock-saml-metadata
        secretKeyName: metadata.xml
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "manual"

lensesHq:
  auth:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    administrators:
      - admin@example.com
      - admin2@example.com
    saml:
      baseURL: ""
      entityID: ""
      metadata:
        referenceFromSecret: false
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          ...
          ...
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"

lensesHq:
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    accessControlAllowCredentials: false
    secureSessionCookies: true
    tls:
      enabled: true
      cert:
      privateKey:
        secret:
          name: 
          key:

lensesHq:
  agents:
    # which port to listen on for agent requests
    address: ":10000" 
    tls:
      enabled: false
      verboseLogs: false
      cert:
      privateKey:

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        referenceFromSecret: true
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        stringData: |
          -----BEGIN CERTIFICATE-----
          ...
          ...
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

lensesHq:
  license:
    referenceFromSecret: true
    secretName: hq-license
    secretKeyName: key

lensesHq:
  licence:
    referenceFromSecret: false
    stringData: "license_key_*"

lensesHq:
  metrics:
    prometheusAddress: ":9090"

(Optional) Configure Ingress & Services

Ingress and service resources are optionally supported.

Enable an Ingress resource in the values.yaml:

ingress:
  http:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
    host: example.com

  agent:
    enabled: true
    agentIngressConfig:
      apiVersion: traefik.containo.us/v1alpha1
      kind: IngressRouteTCP
      metadata:
        name: agents
      spec:
        entryPoints:
          - agents
        routes:
          - match: HostSNI(`example.com`)  # HostSNI to match TLS for TCP
            services:
              - name: lenses-hq            # Replace with your service name
                port: 10000                # Agent default TCP port  
        tls: {}

Enable a service resource in the values.yaml:

# Lenses HQ service
service:
  enabled: true
  type: ClusterIP
  annotations: {}
  externalTrafficPolicy:
  loadBalancerIP: 130.211.x.x
  loadBalancerSourceRanges:
    - 0.0.0.0/0

(Optional) Configure Service Accounts

Lenses HQ, by default, uses default Kubernetes service account but user can choose to use specific one.

If user defines following:

# serviceAccount is the Service account to be used by Lenses to deploy apps
serviceAccount:
  create: true
  annotations: {}
  name: lenses-hq

The chart will create new service account in the the defined namespace for HQ to use.

(Optional) Enable RBAC

There are two options user can choose between:

1. rbacEnable: true - will enable creation of ClusterRole and ClusterRoleBinding for service account mentioned in snippet above

2. rbacEnable: true and namespaceScope: true - will enable creation of Role and RoleBinding which is more restrictive.

Configure logging

There are different logging modes and levels that can be adjusted.

lensesHq:
  logger:
    # Allowed values are: text | json
    mode: "text"

    # Allowed values are: info | debug
    level: "info"

Add chart repository

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio-preview https://lenses.jfrog.io/artifactory/api/helm/helm-charts-preview
helm repo update

Installing HQ

helm install lenses-hq lensesio-preview/lenses-hq \
   --values values.yaml \
   --create-namespace --namespace lenses-hq \
   --version 6.0.0-alpha.14

resources:
  requests:
  #   cpu: 1
    memory: 4Gi
  limits:
  #   cpu: 2
    memory: 8Gi

image:
  repository: lensesio/panoptes:v6.0.0

rbacEnable: true
namespaceScope: true

ingress:
  enabled: true
  annotations: {}
  host: [LENSES_HQ_URL]
  tls:
    enabled: true
    secretName: ingress-cert-secret

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        referenceFromSecret: true
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  api:
    address: ":8080"
    accessControlAllowOrigin: '["https://[LENSES_HQ_URL]"]'
    administrators: '["initial_administrator@email.com"]'
    saml:
      baseUrl: "https://[LENSES_HQ_URL]"
      entityId: "https://[LENSES_HQ_URL]"
      metadata:
        referenceFromSecret: true
        secretName: metadata-secret
        secretKeyName: metadata.xml
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
    tls:
      enabled: false
  # Find more details in https://docs.lenses.io/current/installation/kubernetes/helm/#helm-storage
  ## Postgres template example: "postgres://[username]:[pwd]@[host]:[port]/[database]?sslmode=require"
  storage:
    postgres:
      enabled: true
      host: [POSTGRES_HOST_URL]
      port: 5432
      username: [POSTGRES_USERNAME]
      database: [POSTGRES_USER_PWD]
      passwordSecret:
        type: "precreated"
        name: initContainer-2-db-secret
        key: password

Lenses can be deployed in the following ways:

The Agent docker image can be configured via environment variables or via volume mounts for the configuration files.

Running the Docker

File configuration

In order for command above provisioning yaml has to be configured.

There are two mandatory connections:

1. HQ which requires AgentKey (LENSESHQ_AGENT_KEY) - this key is being created once user registers "New environment" in HQ;

2. Kafka connection

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses.conf can be mounted directly as

• /mnt/settings/lenses.conf

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

• /data/storage

• /data/plugins

• /data/logs

• /data/kafka-streams-state

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Agent upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Agent may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Agent TLS and Global JVM Trust Store

By default, the the serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

1. Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets

2. Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting the Agent.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.

To install the HQ from the archive you must:

1. Extract the archive

2. Configure the HQ

3. Start the HQ

Extracting the archive

Extract the archive using the following command

Inside the extract archive, you will find.

Configuring the HQ

In order to properly configure HQ, two core components are necessary:

• Postgres database

Final Configuration File

If you have meticulously followed all the outlined steps, your config.yaml file should mirror the example provided below, fully configured and ready for deployment. This ensures your system is set up correctly with all necessary settings for authentication, database connection, and other configurations optimally defined.

Starting the HQ

Start Lenses by running:

or pass the location of the config file:

If you do not pass the location of the config file, the HQ will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop HQ, press CTRL+C.

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

The HQ docker image can be configured via volume mounts for the configuration file.

Running the Docker

Prerequisites

The main pre-requirements that has to be fulfilled before Lenses HQ container can be started and those are:

• Postgres database

Complete configuration file

Main configuration file that has to be configured before running docker command is config.yaml.

Sample configuration file is following:

Users can authentication is two ways. Basic authentication and SSO / SAML. Additional specific users can be assigned as admin accounts.

Prerequisites

• Kubernetes 1.23+

• Helm 3.8.0+

• Running Postgres instance

• (in case of ExternalSecret usage)

Install the Chart

First, add the Helm Chart repository using the Helm command line:

Installing the Agent

Installing using cloned repository:

Installing using Helm repository:

Provisioning

To automatically provision the connections to Kafka, HQ and other systems set the .Values.lenses.provision.connections to be the YAML definition of your connections.

The chart will render the full YAML specified under this setting as the provisioning.yaml file.

Alternatively you can use a second YAML file, which contains only the connections pass them at the command line when installing:

Understanding the Chart

The chart uses:

1. Secrets to store Postgres credentials and authentication credentials

2. Secrets to store connection credentials such as Kafka SASL_SCRAM password or password for SSL JKS stores.

3. Secrets to hold the base64 encoded values of the JKS stores

4. Secrets to store AGENT KEY for connection to Lenses HQ

5. ConfigMap for Lenses configuration overrides

6. Cluster roles and role bindings (optional).

Secrets and config maps are mounted as files under the mount /mnt:

1. settings - holds the lenses.conf

2. provision-secrets - holds the secrets for connections in the provisioning.yaml file

3. provision-secrets/files - holds any file needed for a connection, e.g. JKS files.

Connection to Lenses HQ

Connection to Lenses HQ is straight forward process which requires two steps:

2. Storing that same key in Vault or as a K8s secret

The agent communicates with HQ via a secure custom binary protocol channel. To establish this channel and authenticate the Agent needs and AGENT KEY.

Once the AGENT KEY has been copied, store it inside of Vault or any other tool that has integration with Kubernetes secrets.

There are three available options how agent key can be used:

1. ExternalSecret via External Secret Operator (ESO)

2. Pre-created secret

3. Inline string

This secret will be fed into the provisioning.yaml. The HQ connection is specified at line 30, below, where reference ${LENSESHQ_AGENT_KEY} is being set:

Cluster RBAC

The Helm chart creates Cluster roles and bindings, that are used by SQL Processors, if the deployment mode is set to KUBERNETES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.

To disable the creation of Kubernetes RBAC set: rbacEnabled: false

If you are not using SQL Processors and want to limit permissions given to Agent's ServiceAccount, there are two options you can choose from:

1. rbacEnable: true - will enable the creation of ClusterRole and ClusterRoleBinding for service account mentioned above;

2. rbacEnable: true and namespaceScope: true - will enable creation of Role and RoleBinding which is more restrictive.

Postgres

To use Postgres as the backing store for the Agent set the details in the lenses.storage.postgres object.

External Secrets

The chart relies on secrets for sensitive information such as Passwords. Secrets can rotate and are commonly stored in an external store such as Azure KeyVault, Hashicorp Vault or AWS Secrets Manager.

For Postgres, add additional ENV variables via the lenses.additionalEnv object to point to your secret and set the username and password to external in the Postgres section.

Services

Enable a service resource in the values.yaml:

Controlling resources

To control the resources used by the Agent:

Enabling SQL Processors in K8s mode

To enable SQL processor in KUBERENTES mode and control the defaults:

SQL Processor Role Binding

To achieve you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to.

For example:

• Lenses namespace = lenses-ns

• Processor namespace = lenses-proc-ns

Finally you need to define in the Agent configuration which namespaces the Agent has access to. Amend values.yaml to contain the following:

Prometheus metrics

Prometheus metrics are automatically exposed on port 9102 under /metrics.

lenses.conf

The main configurable options for lenses.conf are available in the values.yaml under the lenses object. These include:

• Authentication

• Database connections

• SQL processor configurations

To apply other static configurations use lenses.append.conf, for example:

Example Values files

To install the Agent from the archive you must:

1. Extract the archive

2. Configure the Agent

3. Start the Agent

Extracting the archive

Extract the archive using the following command

Inside the extract archive, you will find.

Configure the Agent

Once the agent files are configure you can continue to start the agent.

Provisioning

Starting the Agent

Start Lenses by running:

or pass the location of the config file:

If you do not pass the location of lenses.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the lenses.conf to be readable only by the lenses user.

The agent needs write access in 4-5 places in total:

1. [RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:

   1. [RUNTIME DIRECTORY]/logs Where logs are stored

   2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.

   3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.

   4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.

   5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

1. Mount /tmp without noexec

2. or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Increase as a super-user the soft limit to 4096 with:

Use 8GB RAM /4 CPUs and 20GB disk space.

You can configure a list of the principals (users, service accounts) that have root admin access. Access control allows any API operation performed by such principals. If not set, it will default to [].

Admin accounts are set in the config.yaml for HQ under the key, as an array of usernames.

Basic authentication is set in the config.yaml for HQ under the key, as an array of usernames and passwords.