1 of 100

6.0-preview

Overview

Welcome to Lenses, Autonomy in data streaming.

This documentation is for Lenses 6 (preview). For Lenses 5.5 (stable) see here.

How Lenses Works

Lenses has two components:

HQ

HQ is a central portal where end users interact with different environments (clusters). It provides a central place to explore data across many environments.

HQ is a single binary, installed on premise or in your cloud. From HQ you create environments, and for each environment, you deploy an agent that connects back to HQ.

Environments

Lenses defines each Kafka Cluster and supporting services, such as Schema Registries and Kafka Connect Clusters, as an environment.

You can have many environments, on premise, in the cloud, provided HQ has network access to the agent and the agent can connect to your Kafka cluster or any Kafka API compatible service.

Each environment has an agent. Environments can also be assigned extra metadata such as tiers, domains and descriptions.

Agents

There's a 1 to 1 relationship between environments, agents and Kafka clusters.

To explore and operate in an environment you need an agent. Agents are headless applications, deployed with connectivity to your Kafka cluster and supporting services.

Agents only ever communicate with HQ, using an Agent key over a secure channel. You can not, as a user, interact directly with them. End users are unaware of agents, only environments.

Agents require:

Agent Key to establish a communication channel to HQ
Connectivity to a Kafka cluster and credentials to do so.

The agents acts as a proxy to read, or write to your Kafka cluster, execute queries, monitor for alerts and manage SQL Processors and Kafka Connectors.

What's New?

This page details the release notes of Lenses.

6.0 Preview

New

Lenses 6.0 introduces a new service, called HQ, acting as portal for multi-kafka environments.

New HQ service
IAM (Identity & Access Management). This has moved from each Lenses instant to a global location in the new HQ service
Global SQL Studio
Global Data Catalogue
Community License: You can now use Lenses without a license (community license key is bundled in the docker-compose) or expiry but the following restrictions apply:
- No SSO
- Maximum of two environments (Kafka clusters) can be connected
- Two Users with one an admin user
- Two Service Accounts
- Two Groups
- Two Roles
- No Backup / Restore for topics to S3

Breaking

H2 embedded database is no longer support.
Lenses 5.x permission model is replaced by global IAM. You must recreate the roles and groups in HQ
Connection management in the agent is via file Provisioning only.

Version 6.0.0-alpha.20

Packages

We have made new alpha release 20:

Agent image:
- lensting/lenses-agent:6.0.0-alpha.6
HQ image:
- lensting/lenses-hq:6.0.0-alpha.20
HQ CLI image
- lensting/lenses-cli:6.0.0-alpha.20

New Helm version 20 for agent and for the HQ: https://lenses.jfrog.io/ui/native/helm-charts-preview/
Archive installation: https://archive.lenses.io/pub/testing/lenses/

Features / Improvements & Fixes

HQ

New features

1. Control how you map SSO groups (including underscores!)

You can freely map a Group to any SSO group including any characters in the name.

Before, a Group was mapped to SSO via its resource-name . That was limiting you to SSO groups with only characters and -dashes.
Now 2 things happen:
- By default, the SSO name is the Group name (not the resource-name). This is most of the time what you'd expect.
- For the special cases when your SSO name is something specific or cryptic (e.g. a UUID) you can override the mapping by setting the SSO mapping nameto anything you want.

2. IAM: permissions for global topic rules

Before, you could set configuration rules for your topics for each of your environments. E.g. enforce topic naming conventions (only dashes) or a maximum number of partitions.

Now, you can control who can access and set these rules with IAM permissions.

The resource type is governance:rule.

Here's an example for read/write access to this rules for environment eu-stg-env:

- effect: allow
  action: 
    - governance:GetRuleDetails
    - governance:UpdateRule
  resource: governance:request:eu-stg-env/*

Find more information in the IAM section.

3. Global SQL Studio: see execution details

See what's happening under the hood of your SQL queries. Learn about your query's performance:

Which partitions were read and how much of them.
How many records were scanned, skipped and offered as results.
Timing and size.
Configuration details that you can tweak.

4. Global SQL Studio: see bad records

The Global SQL Studio will now show you any bad records it cannot understand. These records may be of incorrect formats (e.g. String in an AVRO topic) or have invalid schemas.

Improvements

IAM permissions editor: improved IntelliSense
- For a more IDE-like experience. Get tab completion specific to each segment that you're working on.
Global SQL Studio: improved performance.

Fixes

SSO: fixed offboarding SSO users. When a user has no SSO groups that map to Lenses, Lenses ensures that the user also has no Lenses groups. This is useful when offboarding users to ensure that they won't have Lenses access.
Global SQL Studio: fixed syntax highlighting.

Helm

HQ Changelog

Small bugfixes

Service account annotations were wrongly referenced and were not taking into account upon creation of a new service account
In case authenSignReq was enabled and secrets were not placed it would create environment variable with null values - now it skips creation of environment variables

Agent changelog

Improvements

Breaking change

External secret supports now:

Changeable External Secret Store type: SecretStore | ClusterSecretStore hq:

values.yaml

lensesAgent:  
  hq:
    agentKey:
      secret:
        type: "externalSecret"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key
        externalSecret:
          secretStoreRef:
            type: ClusterSecretStore # ClusterSecretStore | SecretStore
            name: [secretstore_name]

additionalSpecs

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.hq-agent-test.svc.cluster.local
      port: 5432
      username: postgres
      database: hq
      passwordSecret:
        type: "externalSecret"
        name: postgres-aurora
        key:  password
        externalSecret:
          additionalSpecs:
            refreshInterval: 12h
          secretStoreRef:
              type: ClusterSecretStore
              name: enjoy3-secrets

Small bugfixes

Service account annotations were wrongly referenced and were not taking into account upon creation of a new service account

Version 6.0.0-alpha.19

Packages

We have made new alpha release 19:

Agent image:
- lensting/lenses-agent:6.0.0-alpha.5
HQ image:
- lensting/lenses-hq:6.0.0-alpha.19
HQ CLI image
- lensting/lenses-cli:6.0.0-alpha.19

New Helm version 19 for agent and for the HQ: https://lenses.jfrog.io/ui/native/helm-charts-preview/

Features / Improvements & Fixes

HQ

New features

Permissions IntelliSense
New admin permissions to control who can access Lenses Logs

name: nologs
policy:
  - action: administration:GetLensesLogs
    resource: administration:lenses-logs:*
    effect: deny

SQL Studio redesigned view
- scanned partitions progress
- additional statistics
- split tab left and right for SQL queries to compare data.

Improvements

Policy changes are now effective as soon as they are saved, re-login is not necessary anymore

Helm

HQ Changelog

Added support for certificate / signature verification

In case SAML IdP requires certificate verification, same can be now provided.

values.yaml

lensesHq:
  auth:
    saml:
      authnRequestSignature:
        enabled: false
        authnRequestSigningCert:
          referenceFromSecret: true
          secretName: hq-agent-test-authority
          secretKeyName: hq-tls-test.crt.pem
        authnRequestSigningKey:
          secret:
            name: saml-test
            key: privatekey.key

values.yaml

lensesHq:
  auth:
    saml:
      authnRequestSignature:
        enabled: false
        authnRequestSigningCert:
          stringData: |
            -----BEGIN CERTIFICATE-----
            ....
            -----END CERTIFICATE-----
        authnRequestSigningKey:
          secret:
            name: saml-test
            key: privatekey.key

Small bugfixes

HQ on AutnRequest does not send self-signed certificate to avoid validation issues

Version 6.0.0-alpha.18

Packages

We have made new alpha release 18:

Agent image:
- lensting/lenses-agent:6.0.0-alpha.4
HQ image:
- lensting/lenses-hq:6.0.0-alpha.18
HQ CLI image
- lensting/lenses-cli:6.0.0-alpha.18

New Helm version 18 for agent and for the HQ: https://lenses.jfrog.io/ui/native/helm-charts-preview/

Features / Improvements & Fixes

HQ

New features

Agent Metrics are available in overview screen
Introduction of Lenses Resource name in the HQ screens
Added Role description

Improvements

"Roles" panel improved by getting right side panel, detailed view
Dark mode coloring improved

Fixes

Authentication error flashing

Agent

New features

Introduction of Lenses Resource name key resources like: topic, consumers, schemas
Added "Admin Overview" in the Agent menu

Improvements

Fixes

Fixed bad records view

Helm

HQ Changelog

Support for ESO SecretStore

In the past only ClusterSecretStore when handling external secret was support. With a new version you can choose to pick either ClusterSecretStore or a SecretStore when creating external secret.

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres.host.url
      port: 5432
      username: postgres
      database: hq
      passwordSecret:
        type: "externalSecret"
        name: postgres
        key:  password
        externalSecret:
          secretStoreRef:
              type: ClusterSecretStore
              name: css-secrets

values.yaml

lensesHq:  
  storage:
    postgres:
      enabled: true
      host: postgres.host.url
      port: 5432
      username: postgres
      database: hq
      passwordSecret:
        type: "externalSecret"
        name: postgres
        key:  password
        externalSecret:
          secretStoreRef:
              type: SecretStore
              name: ss-secrets

ESO additional specs

New property has been added to external secret template called additionalSpecs where you can add or change any of the specs that would normally be added in ExternalSecret resource.

Example:

lensesHq:  
  storage:
    postgres:
      enabled: true
      host: postgres.host.url
      port: 5432
      username: postgres
      database: hq
      passwordSecret:
        type: "externalSecret"
        name: postgres
        key:  password
        externalSecret:
          additionalSpecs:
            refreshInterval: 12h
          secretStoreRef:
              type: ClusterSecretStore
              name: css-secrets

Small bugfixes

lensesHq.http.tls property was referencing wrong properties within defined values.yaml

Version 6.0.0-alpha.17

Features / Improvements & Fixes

HQ

New features

LRN UX for HQ IAM.
- See & copy the LRN of an IAM resource (User, Group, Service Account, Role).
- LRN is also an available column in all IAM listing pages. You can enable it in the column selector of the table.
Environments + metrics columns.
Tables UX update: powerful grid
- Multi sort
- Reorder of columns + column selection
- Filters
- Preferences are saved for the user
- CLI
  - Provide shell completion for agent --env flag

Improvements

IAM - Updated permissions syntax.
- Renaming. Some services and resources have been renamed. This is to simplify and make the IAM model more consistent. You can find the latest spec here. You may need to adjust your policy YAML definitions. Reach out to us if you need help.
- Prefix/infix wildcard support. You can now use * in a prefix or infix position for resource-id path segments. Use this to express things like "environments that end in -dev" (*-dev) or "topics whose name starts with fraud and ends with analytics" (fraud*analytics).
Improved Global SQL studio UX
- Topic navigator IDE experience.
- See latest topic messages by default, ordered by time, with latest messages 1st.
Dark mode for sign-in page

Fixes

User Profile page now correctly save changes.

Agent

New features

Expose Agent dashboard metrics via API.

Improvements

Improved Agent-level navigation bar.

Fixes

Helm

Packages

We have made new alpha release 17:

Agent image:
- lensting/lenses-agent:6.0.0-alpha.3
HQ image:
- lensting/lenses-hq:6.0.0-alpha.17
HQ CLI image
- lensting/lenses-cli:6.0.0-alpha.17

New Helm version 17 for agent and for the HQ: https://lenses.jfrog.io/ui/native/helm-charts-preview/

HQ Changelog

Support for additional environment variables

When working on software projects, there often arises a need to create additional environment variables for various purposes. One common scenario for this is when users need to securely handle sensitive information, such as passwords or API keys. By storing a user password in a secret, the system ensures that such sensitive information is not exposed to unauthorized access, and this practice offers enhanced security.

values.yaml

lensesHq:
  # Additional env variables appended to deployment
  # Follows the format of [EnvVar spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#envvar-v1-core)
  additionalEnv:
    - name: ADMIN_USER
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-username
    - name: ADMIN_USER_PWD
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-password

values.yaml

lensesHq:
  auth:
    users:
      - username: $(ADMIN_USER)
        password: $(ADMIN_USER_PWD)
  additionalEnv:
    - name: ADMIN_USER
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-username
    - name: ADMIN_USER_PWD
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-password

Rest port consolidation

Property restPorthas been removed and replaced by lensesHq.http.address

Small bugfixes

Agent changelog

Syntax changes

In the provisioning, there has been slight adjustment in the parent agent configuration parameter.

Changes:

lenses has been renamed to lensesAgent

values.yaml

lenses:
  provision:
    enabled: true
    version: "2"
    path: /mnt/provision-secrets
    connections:
      grpcServer:     # Property that has changed
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            apiKey:     # Property that has changed
              value: ${LENSESHQ_AGENT_KEY}

values.yaml

lensesAgent:   # Renamed property
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            agentKey:   
              value: ${LENSESHQ_AGENT_KEY}

Provisioning changes

As provisioning with the latest version (2) is mandatory for successful running of the agent, both configs are removed.

values.yaml

lensesAgent:   # Renamed property
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            agentKey:   
              value: ${LENSESHQ_AGENT_KEY}

Removal of H2 database support

In the the past it was possible to use H2 database which would be instantly deployed and ready to use alongside the agent.

Due to certain performance limitations which come with H2 database which can impact the agent functionality, we decided to completely remove H2 support.

However, persistence parameter still remains and can be used to enable extra volume creation dedicated specifically just logs.

persistence:
  log:
    # -- (boolean) Extra volume creation dedicated for logs.
    # @section -- Persistence scope values
    enabled: true
    # -- (boolean) Annotations dedicated for logs.
    # @section -- Persistence scope values
    annotations: {}
    # -- (list) Access mode rights for created persistence volumes.
    # @section -- Persistence scope values
    accessModes:
      - ReadWriteOnce
    # -- (string) Size of persistence that will be created.
    # @section -- Persistence scope values
    size: 5Gi

Version 6.0.0-alpha.16

Packages

We have made new alpha release 16:

Agent image:

public.ecr.aws/q8a6e1s5/public-agent:v6.0.0-alpha.1-8-g22f83c3e4

lenses/public-agent:v6.0.0-alpha.1-8-g22f83c3e4

HQ image:

public.ecr.aws/q8a6e1s5/public-hq:v6.0.0-alpha.16

```
lenses/public-hq:v6.0.0-alpha.16
```

New Helm version 16 for agent and for the HQ: https://lenses.jfrog.io/ui/native/helm-charts-preview/

HQ Changelog

Introducing License

With the new version of HQ, we are introducing licence. Every customer will receive licence separately.

Additional field acceptEULA has been introduced as well and has to be accepted otherwise HQ will fail on startup.

values.yaml

license:
  # -- (string) Enables usage of secret for licence.
  # **Required: false**
  referenceFromSecret: false
  # -- (string) Secret name where licence is stored.
  # **Required: false**
  secretName: ""
  # -- (string) Secret key where within a secret where licence is sotred.
  # **Required: false**
  secretKeyName: ""
  # Marks the end-user license agreement (EULA) as accepted.
  acceptEULA: true

values.yaml

license:
  stringData: ""
  acceptEULA: true

New authentication method (Password based)

In the new release, password-based authentication has been introduced as an optional method alongside SAML / SSO.

lensesHq_
  auth:
    # -- Adds uses for password based auth
    # **Required: false**
    users:
      - username: admin
        # bcrypt("changeme").
        password: $2a$12$dTSwP3jgCQPoBNDYXNoLy.6l7fMcHYgonl0u8GYCOrkfGM4a.8jze

Existing property samlnow has new field saml.enabled which either enabled or disables SAML / SSO

values.yaml

lensesHq:
  auth:
    saml:
      # -- Enables SAML / SSO authentication
      # **Required: true**
      enabled: false

values.yaml

lensesHq:
  auth:
    administrators:
      - admin@example.com
      - admin
    users:
      - username: admin
        # bcrypt("admin").
        password: $2a$10$DPQYpxj4Y2iTWeuF1n.ItewXnbYXh5/E9lQwDJ/cI/.gBboW2Hodm
    sessionDuration: "23h"
    saml:
      enabled: true
      baseURL: "https://your.hq.url"
      entityID: "https://your.hq.url"
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: true
        secretName: hq-saml-metadata
        secretKeyName: metadata.xml

SAML / SSO is now optional

In previous versions, SAML / SSO was a mandatory requirement for authentication. However, with the new release, it becomes optional, allowing you to choose between password-based authentication and SAML / SSO according to your needs.

Existing alpha users will have to introduce lensesHq.saml.enabled property into their values.yaml files

values.yaml

lensesHq:
  auth:
    saml:
      # -- Enables SAML / SSO authentication
      # **Required: true**
      enabled: false

Ingress structure changes + new agent ingress

In this release, the ingress configuration has been enhanced to provide more flexibility.

Previously, the HQ chart supported a single ingress setting, but now you can define separate ingress configurations for HTTP and the agent.

This addition allows you to tailor ingress rules more specifically to your deployment needs, with dedicated rules for handling HTTP traffic and TCP-based agent connections.

The http ingress is intended only for HTTP/S traffic, while the agents ingress is designed specifically for TCP protocol. Ensure appropriate ingress configuration for your use case.

values.yaml

ingress:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.middlewares: common-traefik-basic-auth@kubernetescrd
  enabled: true
  host: example.com

In the following example you will notice how ingress configuration has been broken into:

http - which covers main ingress for HQ and where users will be accessing HQ portal
agent - new and additional ingress which allows you to add new ingress with your custom implementation, whether it is Traefik or any other based.

By default both http and agent ingresses are disabled.

values.yaml

ingress:
  http:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
    host: example.com

  agent:
    enabled: true
    agentIngressConfig:
      apiVersion: traefik.containo.us/v1alpha1
      kind: IngressRouteTCP
      metadata:
        name: agents
      spec:
        entryPoints:
          - agents
        routes:
          - match: HostSNI(`example.com`)  # HostSNI to match TLS for TCP
            services:
              - name: lenses-hq            # Replace with your service name
                port: 10000                # Agent default TCP port  
        tls: {}

Agent

Due to new changes in provisioning structure, the database to which agent is connected must be recreated.

Changes in provisioning connection to HQ

In the provisioning, there has been slight adjustment in connection naming with HQ.

Changes:

grpcServer has been renamed to lensesHq
apiKey has been renamed to agentKey

values.yaml

lenses:
  provision:
    enabled: true
    version: "2"
    path: /mnt/provision-secrets
    connections:
      grpcServer:     # Property that has changed
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            apiKey:     # Property that has changed
              value: ${LENSESHQ_AGENT_KEY}

values.yaml

lenses:
  provision:
    enabled: true
    version: "2"
    path: /mnt/provision-secrets
    connections:
      lensesHq:    # Renamed property
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            agentKey:    # Renamed property
              value: ${LENSESHQ_AGENT_KEY}

Known issues

With the new version of Agent, HQ connection in provisioning has changed which requires complete recreation of database. Following log message will indicate it:

 liquibase.exception.ValidationFailedException: Validation Failed:                                                                                                           │
│      1 change sets check sum                                                                                                                                                │
│           io/lenses/store/jdbc/migration/6.0.0/02_new_template_data.xml::2::lenses was: 8:357e3bd4e93a5cc938420eb2521c4b7c but is now: 8:03dad3472f5facacdd10a985e5e02da3   │

Version 6.0.0-alpha.14

HQ Changelog

HQ configuration file TOML -> YAML

In the past, HQ has been using TOML file format. As we want to reduce differences in file formats between Agent and HQ as much as possible, this was the first step.

Postgres changes

Postgres connection URI is not being built within config.yaml but in backend runtime;
- parameter group has changed from postgres to storage.postgres.*
In the previous version, schema was defined as a part of extraParamSpecs. In the new version schema is now defined as a separate property storage.postgres.database.schema;
Property extraParamSpecs is renamed to params;

Parameter group api changes

Parameter group api has been renamed to http and following parameters are not part of it anymore:

administrators;
saml;

New parameter auth

Property auth is being derived from property api (now. http).

Parameters that has been moved from http to auth are following:

administrators;
saml;

Aurora support

HQ has been tested against Aurora (Postgres) and is compatible.

Pod restart on configmap checksum change

In case of any changes in ConfigMap and after executing helm upgrade HQ pod will be automatically restarted as well therefore no need for manual interventions.

TL; DR Changes in values.yaml

values.yaml

lensesHq:
  auth:
    administrators:
      - admin@example.com
      - admin2@example.com
    saml:
      baseURL: "https://"
      entityID: "https://"
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: hq-tls-mock-saml-metadata
        secretKeyName: metadata.xml
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://saml.example.com/entityid" validUntil="2034-09-23T08:10:34.764Z">
            <md:IDPSSODescriptor WantAuthnRequestsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
              <md:KeyDescriptor use="signing">
                <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                  <ds:X509Data>
                    <ds:X509Certificate>MIIC4jCCAcoCCQC33wnybT5QZDANBgkqhkiG9w0BAQsFADAyMQswCQYDVQQGEwJV
          ...
          m0eo2USlSRTVl7QHRTuiuSThHpLKQQ==</ds:X509Certificate>
                  </ds:X509Data>
                </ds:KeyInfo>
              </md:KeyDescriptor>
              <md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mocksaml.com/api/saml/sso"/>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mocksaml.com/api/saml/sso"/>
            </md:IDPSSODescriptor>
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "manual"
    tls:
      enabled: false
      cert:
        referenceFromSecret: false
        stringData: |
          -----BEGIN CERTIFICATE-----
          MIIECDCCAvCgAwIBAgIRAIiph6RQMbGCABOaJO94PbowDQYJKoZIhvcNAQELBQAw
          ...
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  agents:
    address: ":10000"
    tls:
      enabled: false
      verboseLogs: false
      cert:
        referenceFromSecret: false
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
        stringData: |
          -----BEGIN CERTIFICATE-----
          MIIECDCCAvCgAwIBAgIRAIiph6RQMbGCABOaJO94PbowDQYJKoZIhvcNAQELBQAw
          gYUxFTATBgNVBAYTDEFua2gtTW9ycG9yazEaMBgGA1UEChMRVW5zZWVuIFVuaXZl
          .....
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - https://
      - http://
  storage:
    postgres:
      #host: postgres-postgresql.postgres.svc.cluster.local
      host: 
      port: 5432
      #username: lenses
      username: postgres
      database: hq
      passwordSecret:
        type: "externalSecret"
        name: postgres-aurora
        key:  password
        externalSecret:
          secretStoreRef:
            clusterSecretStore:
              name: enjoy3-secrets
  logger:
    mode: "text"
    level: "debug"

  # Additional env variables appended to deployment
  # Follows the format of [EnvVar spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#envvar-v1-core)
  additionalEnv:
    # - name: FOO
    #   value: bar

  # Disable livenessProbe, used while debugging
  livenessProbe:
    enabled: false

  # Pause execution of Lenses start up script to allow the user to login into the container and
  # check the running environment, used while debugging
  pauseExec:
    enabled: false

  monitoring:
    port: 9090

values.yaml

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: false
  api:
    address: ":8080"
    accessControlAllowOrigin: '["http://localhost:8080"]'
    administrators: '["admin@example.com"]'
    saml:
      baseUrl: "http://localhost:8080"
      entityId: "http://localhost:8080"
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://saml.example.com/entityid" validUntil="2034-09-23T08:10:34.764Z">
            <md:IDPSSODescriptor WantAuthnRequestsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
              <md:KeyDescriptor use="signing">
                <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                  <ds:X509Data>
                    <ds:X509Certificate>MIIC4jCCAcoCCQC33wnybT5QZDANBgkqhkiG9w0BAQsFADAyMQswCQYDVQQGEwJV
          SzEPMA0GA1UECgwGQm94eUhRMRIwEAYDVQQDDAlNb2NrIFNBTUwwIBcNMjIwMjI4
          MjE0NjM4WhgPMzAyMTA3MDEyMTQ2MzhaMDIxCzAJBgNVBAYTAlVLMQ8wDQYDVQQK
          DAZCb3h5SFExEjAQBgNVBAMMCU1vY2sgU0FNTDCCASIwDQYJKoZIhvcNAQEBBQAD
          ggEPADCCAQoCggEBALGfYettMsct1T6tVUwTudNJH5Pnb9GGnkXi9Zw/e6x45DD0
          RuRONbFlJ2T4RjAE/uG+AjXxXQ8o2SZfb9+GgmCHuTJFNgHoZ1nFVXCmb/Hg8Hpd
          4vOAGXndixaReOiq3EH5XvpMjMkJ3+8+9VYMzMZOjkgQtAqO36eAFFfNKX7dTj3V
          pwLkvz6/KFCq8OAwY+AUi4eZm5J57D31GzjHwfjH9WTeX0MyndmnNB1qV75qQR3b
          2/W5sGHRv+9AarggJkF+ptUkXoLtVA51wcfYm6hILptpde5FQC8RWY1YrswBWAEZ
          NfyrR4JeSweElNHg4NVOs4TwGjOPwWGqzTfgTlECAwEAATANBgkqhkiG9w0BAQsF
          AAOCAQEAAYRlYflSXAWoZpFfwNiCQVE5d9zZ0DPzNdWhAybXcTyMf0z5mDf6FWBW
          5Gyoi9u3EMEDnzLcJNkwJAAc39Apa4I2/tml+Jy29dk8bTyX6m93ngmCgdLh5Za4
          khuU3AM3L63g7VexCuO7kwkjh/+LqdcIXsVGO6XDfu2QOs1Xpe9zIzLpwm/RNYeX
          UjbSj5ce/jekpAw7qyVVL4xOyh8AtUW1ek3wIw1MJvEgEPt0d16oshWJpoS1OT8L
          r/22SvYEo3EmSGdTVGgk3x3s+A0qWAqTcyjr7Q4s/GKYRFfomGwz0TZ4Iw1ZN99M
          m0eo2USlSRTVl7QHRTuiuSThHpLKQQ==</ds:X509Certificate>
                  </ds:X509Data>
                </ds:KeyInfo>
              </md:KeyDescriptor>
              <md:NameIDFormat>urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress</md:NameIDFormat>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://mocksaml.com/api/saml/sso"/>
              <md:SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://mocksaml.com/api/saml/sso"/>
            </md:IDPSSODescriptor>
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
    tls:
      enabled: false
  # Find more details in https://docs.lenses.io/current/installation/kubernetes/helm/#helm-storage
  ## Postgres template example: "postgres://[username]:[pwd]@[host]:[port]/[database]?sslmode=require"
  postgres:
    host: postgres-postgresql.postgres.svc.cluster.local
    port: 5432
    username: lenses
    database: hq
    #extraParamSpecs: "?sslmode=disable"
    passwordSecret:
      type: "externalSecret"
      name: postgres
      key:  password
      externalSecret:
        secretStoreRef:
          clusterSecretStore:
            name: [CLUSTER_SECRET_STORE_NAME]

  logger:
    mode: "text"
    level: "debug"

  # Additional env variables appended to deployment
  # Follows the format of [EnvVar spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#envvar-v1-core)
  additionalEnv:
    # - name: FOO
    #   value: bar

  # Disable livenessProbe, used while debugging
  livenessProbe:
    enabled: true

  monitoring:
    port: 9090

Agent Changelog

Agent Key reference change

Previously environment variable known as LENSES_HQ_AGENT_KEY that was referenced in provisioning.yaml and stores the agentKey value has been renamed to LENSESHQ_AGENT_KEY.

Known issues

Since newest version of Lenses HQ and Agent bring breaking changes following issues can happen.

Database migration

Upon doing helm upgrade HQ can fail with following error log:

Fatal: Init db error" err="migrate database: database has been migrated beyond what we understand: the current is 17; the target is 2" version=v6.0.0-alpha.14

In order to fix it, following command has to be run on the postgres database:

DELETE FROM goose_db_version WHERE id > 2;

In case SQL command cannot be run, database has to be cleared as if one is starting from scratch.

Getting Started

Quick Start

This quick start guide will walk you through installing and starting Lenses using Docker, followed by connecting Lenses to your Kafka cluster.

This is quick start is for a local setup. To connect to your Kafka clusters, see here.

By running the following command including the ACCEPT_EULA setting your are accepting the Lenses EULA agreement.

Run the following command:

terminal

wget https://lenses.io/preview -O docker-compose.yml \
&& ACCEPT_EULA=true docker compose up

Once the images are pulled and containers started, you can login here with admin/admin and explore.

It may take a few seconds for the agent to full boot and connect to HQ.

The quick start uses a docker compose file to:

Start Postgres, the Agent use Postgres as a backing store.

Start a single local Kafka Broker

Start a local Confluent Schema Registry

Start HQ and create an environment & agent key

Start Agent and to connect to Kafka, the Schema Registry and HQ.

Connecting Lenses to your Kafka environment

Connect Lenses to your environment.

Overview

This page describe an overview of deploying Lenses against your Kafka clusters.

The quick start is for local development, with a local Kafka. This guide takes you through manually deploying HQ and an Agent to connect to your Kafka clusters.

For more detailed guides on the Helm, Docker and Linux see here.

How to connect to your Kafka?

To deploy Lenses against your environments you need to:

Configure HQ

Optionally using your own Postgres instance.

Start HQ

Start HQ using your configuration from step 1.

Create an Environment in HQ

To connect an agent to HQ for your Kafka cluster we need an to create an environment in HQ.

Configure & start an Agent

To configure the agent you need to:

Optionally using your own Postgres instance
Configure a connection to your Kafka cluster and HQ (with the key from step 2)

Prerequisites

EULA acceptance

To start HQ and an Agent you have to accept the Lenses EULA.

For HQ, in the config.yaml set:

config.yaml

license:
  acceptEULA: true

Kafka

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud. Supported providers include:

Confluent Platform & Cloud
AWS MSK & AWS MSK Serverless
Aiven
IBM Event Streams
Azure HDInsight & EventHubs

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

Postgres

Only needed if you want to bring your own Postgres. The docker compose will start a local Postgres instance.

HQ and Agents can share the same instance, by either using a separate database or schema for HQ and each agent, depending on your networking needs.

Postgres server running version 9.6 or higher.

Database Role

The recommended configuration is to create a dedicated login role and database for the HQ and each Agent, setting the HQ or Agent role as the database or schema owner. Both the agent and HQ need credentials, create a role for each.

terminal

# login as superuser and add Lenses role and database
psql -U postgres -d postgres <<EOF
CREATE ROLE lenses_agent WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_agent OWNER lenses_agent;

CREATE ROLE lenses_hq WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_hq OWNER lenses_hq;
EOF

Networking

Web sockets - You may need to adjust your load balancer to allow them. See here.
JMX connectivity - Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and Open Metrics (MSK).

For more enable JMX for Agent itself see here.

Kafka ACLs

These ACLs are for the underlying Lenses Agent Kafka client. Lenses has its own set of permissions guarding access.

You can restrict the access of the Lenses Kafka client but this can reduce the functionality on offer in Lenses, e.g. not allow Lenses to create topic at all, even though this can be managed by Lenses own IAM system.

The agent requires access to your Kafka cluster. If ACLs are enable you will need to allow the Agent access.

SSO (optional)

If you want to use SSO / SAML for authentication you will need the metadata.xml file from your provider. See Authentication for more information.

Install

This page describes configuring and starting Lenses HQ and Agent against your Kafka cluster.

This guide is using the Lenses docker compose file. For non dev installations and automation see the Installation section.

Configure HQ

HQ is configured via by one file, config.yaml. The docker compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

You only need to follow this step if you do not want to use the local postgres instance started by the docker compose file.

You must create a database and role in your postgres instance for HQ to use. See Database Role.

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

docker-compose.yaml

 hq.config.yaml:
    content: |
      # ACCEPT THE LENSES EULA
      license:
        acceptEULA: true
      database:
        host: postgres:5432
        username: [YOUR_POSTGRES_YOUR_NAME]
        password: lenses
        database: hq

Authentication

Currently HQ supports:

Basic Authentication (default)
SAML

For this example we will use basic authentication, for information on configuring other methods, see Authentication and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

terminal

docker-compose up hq

You can now log in your browser with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

Login into HQ and create an environment, Environments->New Environment.
At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

terminal

➜  out ./hq-linux-amd64 environments
Manage Environments.

Usage:
  hq environments [command]

Aliases:
  environments, e, env, envs

Available Commands:
  create      Creates a new environment.
  delete      Deletes an environment.
  get         Retrieves a single environment by name.
  list        Lists all environments
  metadata    Manages environment metadata.
  update      Updates an environment.
  watch       Watch live environment updates.

Configure the Agent

The Agent is configured via two files:

lenses.conf - holds low-level configuration options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file
provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

You only need to follow this step if you do not want to use the local postgres instance started by the docker compose file.

You must create a database and role in your postgres instance for the Agent to use. See Database Role.

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

docker-compose.yaml

 agent.lenses.conf:
    content: |
      lenses.storage.postgres.host=[YOUR_POSTGRES_INSTANCE]
      lenses.storage.postgres.port=[YOUR_POSTGRES_PORT]
      lenses.storage.postgres.database=agent
      lenses.storage.postgres.username=lenses
      lenses.storage.postgres.password=lenses

Adding a HQ Connection

The Agent Key for an environment needs to be added to the agent.provisioning.yaml key in the docker compose file.

docker-compose.yaml

agent.provisioning.yaml:
    content: |
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace ${{LENSESHQ_AGENT_KEY}} with the Agent Key for the environment that you want to link to.

For more information on the configuration of the connection to HQ see here.

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this update the agent.provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

See provisioning for examples of different authentication types for Kafka.

Add the following for a basic plaintext connection to a Kafka broker, if you are using a different authentication mechanism adjust accordingly.
Remove, or adjust the Kafka (kafka-demo), Schema Registry and Connect services in the default docker-compose file.

docker-compose.yaml

agent.provisioning.yaml:
    content: |
      kafka:
        - name: kafka
          version: 1
          tags: [ 'kafka', 'dev' ]
          configuration:
            metricsType:
              value: JMX
            metricsPort:
              value: 9581
            kafkaBootstrapServers:
              value: PLAINTEXT://[YOUR_BOOTSTRAP_BROKER:PORT] 
            protocol:
              value: PLAINTEXT
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace [YOUR_BOOTSTRAP_BROKER:PORT] with the bootstrap brokers and ports for the Kafka cluster you want the Agent to connect to.

For examples of adding in other services such as Schema Registries and Kafka Connect see provisioning.

Start the Agent

To start Agent, run the following docker command:

For none dev environments, install the agent as close as possible to your Kafka clusters and automate the installation.

terminal

docker-compose up agent

Once the agent fully starts, it will report as connected in HQ, allowing you to explore your Kafka environments.

Deployment

Installation

This page describes the supported installation methods for Lenses.

Lenses can be deployed in the following ways:

Kubernetes - Helm

This page describes installing Lenses HQ and Agent in Kubernetes via Helm.

Only Helm 3 is supported.

Deploying HQ

This page describes installing Lenses HQ in Kubernetes via Helm.

Lenses HQ is prerequisite for installation of Lenses Agent

Latest images (v19):

HQ image: lensting/lenses-hq:6-preview
HQ Cli image: lensting/lenses-cli:6-preview

Prerequisites

Kubernetes 1.23+
Helm 3.8.0+
Running Postgres instance:
- database for HQ;
- username (and password) that has access to HQ database;
Optional External secret operator (in case of ExternalSecret usage)

Configure HQ

In order to configure properly HQ, we have to understand parameter groups that the Chart offers.

Under the lensesHq parameter there some key parameter groups that are used to setup HQ:

storage
- definition of connection towards database (Postgres is the only storage option)
auth
- Password based authentication configuration
- SAML / SSO configuration
- definition of administrators or first users to access the HQ
http
- defines port under which HQ will be available for end users
- defines values of special headers and cookies
- types of connection such as TLS and non-TLS definitions
agents
- defines connection between HQ and the Agent such as port where HQ will be listening for agent connections.
- types of connection such as TLS and non-TLS definitions
license
monitoring
- controls the metrics settings where Prometheus alike metrics will be exposed
loggers
- definition of logging level for HQ

Moving forward, in the same order you can start configuring your Helm chart.

Configure storage (Postgres)

Postgres is the only available storage option.

Prerequisite:

Running Postgres instance;
Created database for HQ;
Username (and password) which has access to created database;

In order to successfully run HQ, storage within values.yaml has to be defined first.

Definition of storage object is as follows:

lensesHq:
  storage:
    postgres:
      enabled: true
      host: ""
      port: 
      username: ""
      database: ""
      schema: ""
      tls: 
      params: {}
      passwordSecret:
        type: ""

Alongside Postgres password, which can be referenced / created through Helm chart, there are few more options which can help while setting up HQ.

Username reference types

There are two ways how username can be defined:

The most straight forward way if username is not being changed by just defining it within username parameter such as

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses

In case Postgres username is being rotated or frequently changed it can be referenced from pre-created secret

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses
      useSecretForUsername:
        enabled: true
        existingSecret:
          name: my-secret
          key: username

Password reference types

Postgres password can be handled in three ways using:

External Secret via ExternalSecretOperator;
Pre-created secret;
Creating secret on the spot through values.yaml;

To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying HQ.

When specifying passwordSecret.type: "externalSecret", the chart will:

create an ExternalSecret in the namespace where HQ is deployed;
a secret is mounted for HQ to use.

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.playground.svc.cluster.local
      port: 5432
      username: lenses
      database: lenseshq
      passwordSecret:
        type: "externalSecret"
        # Secret name where database password will be read from
        name: hq-password
        # Key name under secret where database password is stored
        key: password
        externalSecret:
          additionalSpecs: {}
          secretStoreRef:
            type: SecretStore # or ClusterSecretStore
            name: secretstore-secrets

Make sure that secret you are going to use is already created in namespace where HQ will be installed.

values.yaml

 lensesHq:
   storage:  
     postgres:
       enabled: true
       host: postgres-postgresql.playground.svc.cluster.local
       port: 5432
       username: lenses
       database: lenseshq
       passwordSecret:
         type: "precreated"
         # Secret name where database password will be read from
         name: hq-password
         # Key from secret's data where database password is being stored
         key: postgres-password

This option is NOT for PRODUCTION usage but rather just for demo / testing.

The chart will create a secret with defined values below and the same secret will be read by HQ in order to connect to Postgres.

values.yaml

 lensesHq:
   storage:
     postgres:
       enabled: true
       host: [POSTGRES_HOSTNAME]
       port: 5432
       username: lenses
       database: lenseshq
       passwordSecret:
         type: "createNew"
         # name of a secret that will be created
         name: [K8s_SECRET_NAME]
         # Database password
         password: [DATABASE_USER_PASSWORD]

Advanced Postgres settings

Sometimes to form correct connection URI special parameters are needed. In order to od the same you can set extra settings using params.

Example:

values.yaml

lensesHq:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      username: lenses
      params:
        sslmode: require

Configure AUTH endpoint

SAML / SSO is available only with Enterprise license.

Second pre-requirement to successfully run HQ is setting initial authentication.

You can choose between:

password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.

Definition of auth object is as follows:

values.yaml

lensesHq:
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG
    administrators:
      - admin
      - admin@example.com
      - admin2@example.com
    saml:
      enabled: true
      baseURL: ""
      entityID: ""
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: ""
        secretKeyName: ""
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          </md:EntityDescriptor>
  
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
      uiRootURL: "/"
      groupAttributeKey: "groups"

First to cover is users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. The passwords need to be hashed using bcrypt before placed within password property for security purposes, ensuring that they are stored correctly and securely.

values.yaml

lensesHq:
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG

values.yaml

lensesHq:
  auth:
    users:
      - username: $(ADMIN_USER)
        password: $(ADMIN_USER_PWD)
  additionalEnv:
    - name: ADMIN_USER
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-username
    - name: ADMIN_USER_PWD
      valueFrom:
        secretKeyRef:
          name: multi-credentials-secret
          key: user1-password

Second to cover will be administrators. It serves as definition of user emails which will have highest level of permissions upon authentication to HQ.

Third attribute is saml.metadata field needed for setting SAML / SSO authentication. In this step, you will need metadata.xml file which can be set in two ways:

Referencing metadata.xml file through pre-created secret;
Placing metadata.xml contents inline as a string.

lensesHq:
  auth:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    administrators:
      - admin@example.com
      - admin2@example.com
    saml:
      baseURL: ""
      entityID: ""
      metadata:
        referenceFromSecret: true
        secretName: hq-tls-mock-saml-metadata
        secretKeyName: metadata.xml
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "manual"

lensesHq:
  auth:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    administrators:
      - admin@example.com
      - admin2@example.com
    saml:
      baseURL: ""
      entityID: ""
      metadata:
        referenceFromSecret: false
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          ...
          ...
          </md:EntityDescriptor>
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"

In case SAML IdP requires certificate verification, same can be enabled and provided in the following way:

values.yaml

lensesHq:
  auth:
    saml:
      authnRequestSignature:
        enabled: false
        authnRequestSigningCert:
          referenceFromSecret: true
          secretName: hq-agent-test-authority
          secretKeyName: hq-tls-test.crt.pem
        authnRequestSigningKey:
          secret:
            name: saml-test
            key: privatekey.key

values.yaml

lensesHq:
  auth:
    saml:
      authnRequestSignature:
        enabled: false
        authnRequestSigningCert:
          stringData: |
            -----BEGIN CERTIFICATE-----
            ....
            -----END CERTIFICATE-----
        authnRequestSigningKey:
          secret:
            name: saml-test
            key: privatekey.key

Configure HTTP endpoint

Third pre-requirement to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around HTTP endpoint of the HQ itself and how users will interact with.

Definition of HTTP object is as follows:

lensesHq:
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    accessControlAllowCredentials: false
    secureSessionCookies: true
    tls:
      enabled: true
      cert:
      privateKey:
        secret:
          name: 
          key:

Second part of HTTP definition would be enabling TLS and TLS definition itself. As previously defined for lensesHq.agents.tls same way of configuring TLS can be used for lensesHq.http.tls definition as well.

Configure agents connection endpoint

After correctly configuring authentication strategy and connection endpoint , agent handling is the last most important box to tick.

The Agent's object is defined as follows:

lensesHq:
  agents:
    # which port to listen on for agent requests
    address: ":10000" 
    tls:
      enabled: false
      verboseLogs: false
      cert:
      privateKey:

Enabling TLS

By default TLS for the communication between Agent and HQ is disabled. In case requirement is to enabled it, following has to be set:

lensesHq.agents.tls - certificates to manage connection between HQ and the Agents
lensesHq.http.tls- certificates to manage connection with HQ's API

Unlike private keys which can be referenced and obtained only through a secret, Certificates can be referenced directly in values.yaml file as a string or as a secret.

values.yaml

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        referenceFromSecret: true
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem

values.yaml

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true
      cert:
        stringData: |
          -----BEGIN CERTIFICATE-----
          ...
          ...
          -----END CERTIFICATE-----
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem

Configure license

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

License can be read in multiple ways:

from a pre-created secret
directly as a string defined in values.yaml file

values.yaml

lensesHq:
  license:
    referenceFromSecret: true
    secretName: hq-license
    secretKeyName: key

values.yaml

lensesHq:
  licence:
    referenceFromSecret: false
    stringData: "license_key_*"

Configure metrics endpoint

Metrics are optionally available in a Prometheus format and by default served on port 9090.

Port can be changed in a following way:

values.yaml

lensesHq:
  metrics:
    prometheusAddress: ":9090"

(Optional) Configure Ingress & Services

Whilst the chart supports setting TLS on Lenses HQ itself we recommend placing it on the Ingress resource

Ingress and service resources are optionally supported.

The http ingress is intended only for HTTP/S traffic, while the agents ingress is designed specifically for TCP protocol. Ensure appropriate ingress configuration for your use case.

Enable an Ingress resource in the values.yaml:

values.yaml

ingress:
  http:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
    host: example.com
    ingressClassName: ""
    tls:
      enabled: false
      # The TLS secret must contain keys named tls.crt and tls.key that contain the certificate and private key to use for TLS.
      secretName: ""


  agent:
    enabled: true
    agentIngressConfig:
      apiVersion: traefik.containo.us/v1alpha1
      kind: IngressRouteTCP
      metadata:
        name: agents
      spec:
        entryPoints:
          - agents
        routes:
          - match: HostSNI(`example.com`)  # HostSNI to match TLS for TCP
            services:
              - name: lenses-hq            # Replace with your service name
                port: 10000                # Agent default TCP port  
        tls: {}

Enable a service resource in the values.yaml:

values.yaml

# Lenses HQ service
service:
  enabled: true
  type: ClusterIP
  annotations: {}
  externalTrafficPolicy:
  loadBalancerIP: 130.211.x.x
  loadBalancerSourceRanges:
    - 0.0.0.0/0

(Optional) Configure Service Accounts

Lenses HQ, by default, uses default Kubernetes service account but user can choose to use specific one.

If user defines following:

values.yaml

# serviceAccount is the Service account to be used by Lenses to deploy apps
serviceAccount:
  create: true
  annotations: {}
  name: lenses-hq

The chart will create new service account in the the defined namespace for HQ to use.

(Optional) Enable RBAC

There are two options user can choose between:

rbacEnable: true - will enable creation of ClusterRole and ClusterRoleBinding for service account mentioned in snippet above
rbacEnable: true and namespaceScope: true - will enable creation of Role and RoleBinding which is more restrictive.

Configure logging

There are different logging modes and levels that can be adjusted.

values.yaml

lensesHq:
  logger:
    # Allowed values are: text | json
    mode: "text"

    # Allowed values are: info | debug
    level: "info"

Add chart repository

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio-preview https://lenses.jfrog.io/artifactory/api/helm/helm-charts-preview
helm repo update

Installing HQ

Be aware that for the time being and for alpha purposes usage of --versionis mandatory when deploying Helm chart through Helm repository.

terminal

helm install lenses-hq lensesio-preview/lenses-hq \
   --values values.yaml \
   --create-namespace --namespace lenses-hq \
   --version 6.0.0-alpha.19

Example of values.yaml

More about default values for Lenses HQ Helm Chart can be found in values.yaml. An example is below:

values.yaml

resources:
  requests:
  #   cpu: 1
    memory: 4Gi
  limits:
  #   cpu: 2
    memory: 8Gi

image:
  repository: lensesio/lenses:latest

rbacEnable: true
namespaceScope: true

ingress:
  http:
    enabled: true
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: websecure
    host: example.com

lensesHq:
  agents:
    address: ":10000"
    tls:
      enabled: true. # optional
      cert:
        referenceFromSecret: true
        secretName: hq-agent-test-authority
        secretKeyName: hq-tls-test.crt.pem
      privateKey:
        secret:
          name: hq-agent-test-authority
          key: hq-tls-test.key.pem
  auth:
    users:
      - username: admin
        # bcrypt("correcthorsebatterystaple").
        password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG
    administrators:
      - admin
      - admin@example.com
      - admin2@example.com
    saml:
      enabled: true    # optional
      baseURL: ""
      entityID: ""
      # -- Example: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>
      metadata:
        referenceFromSecret: false
        secretName: ""
        secretKeyName: ""
        stringData: |
          <?xml version="1.0" encoding="UTF-8" standalone="no"?>
          </md:EntityDescriptor>
  
      userCreationMode: "sso"
      usersGroupMembershipManagementMode: "sso"
      uiRootURL: "/"
      groupAttributeKey: "groups"
  http:
    address: ":8080"
    accessControlAllowOrigin:
      - 
    accessControlAllowCredentials: false
    secureSessionCookies: true
    tls:
      enabled: true    # optional
      cert:
      privateKey:
        secret:
          name: 
          key:
  # Find more details in https://docs.lenses.io/current/installation/kubernetes/helm/#helm-storage
  ## Postgres template example: "postgres://[username]:[pwd]@[host]:[port]/[database]?sslmode=require"
  storage:
    postgres:
      enabled: true
      host: [POSTGRES_HOST_URL]
      port: 5432
      username: [POSTGRES_USERNAME]
      database: [POSTGRES_USER_PWD]
      passwordSecret:
        type: "precreated"
        name: initContainer-2-db-secret
        key: password

Whats' next?

After successful configuration and installation of HQ the next steps would be:

Deploying an Agent

This page describes installing Lenses Agent in Kubernetes via Helm.

Latest Agent image lensting/lenses-agent:6-preview (v4)

Prerequisites

Kubernetes 1.23+
Helm 3.8.0+
Running Postgres instance
External secret operator (in case of ExternalSecret usage)

Configure an Agent

In order to configure properly an Agent, we have to understand parameter groups that the Chart offers.

Under the lensesAgent parameter there some key parameter groups that are used to setup HQ:

Storage
HQ connection
Provision
Cluster RBACs

Moving forward, in the same order you can start configuring your Helm chart.

Configuring Agent chart

Configure storage (Postgres)

Postgres is the only available storage option.

Prerequisite:

Running Postgres instance;
Created database for an Agent;
Username (and password) which has access to created database;

In order to successfully run HQ, storage within values.yaml has to be defined first.

Definition of storage object is as follows:

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: ""
      port: 
      username: ""
      database: ""
      schema: ""
      params: {}

Alongside Postgres password, which can be referenced / created through Helm chart, there are few more options which can help while setting up HQ.

There are two ways how username can be defined:

The most straight forward way if username is not being changed by just defining it within username parameter such as

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: lenses

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: external  # use "external" to manage it using secrets
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_USERNAME
      valueFrom:
        secretKeyRef:
          name: [SECRET_RESOURCE_NAME]
          key: [SECRET_RESOURCE_KEY]

Password reference types

Postgres password can be handled in three ways using:

Pre-created secret;
Creating secret on the spot through values.yaml;

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.playground.svc.cluster.local
      port: 5432
      username: lenses
      database: lensesagent
      password: useOnlyForDemos

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: postgres-postgresql.postgres.svc.cluster.local
      port: 5432
      database: lensesagent
      username: lenses
      password: external   # use "external" to manage it using secrets
  additionalEnv:
    - name: LENSES_STORAGE_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: [SECRET_RESOURCE_NAME]
          key: [SECRET_RESOURCE_KEY]

Configure HQ connection (agent key)

Connection to Lenses HQ is straight forward process which requires two steps:

Creating Environment and obtaining AGENT KEY in HQ as described here, if you already have not done so.
Storing that same key in Vault or as a K8s secret

The agent communicates with HQ via a secure custom binary protocol channel. To establish this channel and authenticate the Agent needs and AGENT KEY.

Once the AGENT KEY has been copied, store it inside of Vault or any other tool that has integration with Kubernetes secrets.

There are three available options how agent key can be used:

ExternalSecret via External Secret Operator (ESO)
Pre-created secret
Inline string

To use this option, the External Secret Operator (ESO) has to be installed and available for usage in K8s cluster your are deploying Agent.

When specifying secret.type: "externalSecret", the chart will:

create an ExternalSecret in the namespace where Agent is deployed;
a secret is mounted for Agent to use.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "externalSecret"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key
        externalSecret:
          additionalSpecs: {}
          secretStoreRef:
            type: ClusterSecretStore # ClusterSecretStore | SecretStore
            name: [secretstore_name]

Make sure that secret you are going to use is already created in namespace where Agent will be installed.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "precreated"
        # Secret name where agentKey will be read from
        name: hq-password
        # Key name under secret where agentKey is stored
        key: key

This option is NOT for PRODUCTION usage but rather just for demo / testing.

The chart will create a secret with defined values below and the same secret will be read by Agent to connect to HQ.

values.yaml

lensesAgent:
  hq:
    agentKey:
      secret:
        type: "createNew"
        # Secret name where agentKey will be read from
        name: "lenses-agent-secret-1"
        # Value of agentKey generated by HQ
        value: "agent_key_*"

This secret will be fed into the provisioning.yaml. The HQ connection is specified at line 30, below, where reference ${LENSESHQ_AGENT_KEY} is being set:

values.yaml

lensesAgent:
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [LENSES_HQ_FQDN_OR_IP]
            port:
              value: 10000
            agentKey:
              # This property shouldn't be changed as it is mounted automatically
              # based on secret choice for hq.agentKey above 
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false

In order to enable TLS for secure communication between HQ and the Agent please refer to the following part of the page.

Configure provisioning (Kafka / SchemaRegistry / Kafka Connect)

Provisioning offers various connection starting with:

Kafka ecosystem components such as:
Alerts & Audits
AWS

values.yaml

lensesAgent:
  provision:
    path: /mnt/provision-secrets
    connections:
      # Kafka Connection
      kafka:
        - name: Kafka
          version: 1
          tags: [my-tag]
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://your.kafka.broker.0:9092
                - PLAINTEXT://your.kafka.broker.1:9092
            protocol: 
              value: PLAINTEXT
            # all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Confluent Schema Registry Connection
      confluentSchemaRegistry:
        - name: schema-registry
          tags: ["tag1"]
          version: 1      
          configuration:
            schemaRegistryUrls:
              value:
                - http://my-sr.host1:8081
                - http://my-sr.host2:8081
            ## all metrics properties are optional
            metricsPort: 
              value: 9581
            metricsType: 
              value: JMX
            metricsSsl: 
              value: false
      # Kafka Connect connection
      connect:
        - name: my-connect-cluster-name
          version: 1    
          tags: ["tag1"]
          configuration:
            workers:
              value:
                - http://my-kc.worker1:8083
                - http://my-kc.worker2:8083
            metricsPort: 
              value: 9585
            metricsType: 
              value: JMX

More about provisioning and more advanced configuration options for each of these components can be found on the following link.

Cluster RBACs

The Helm chart creates Cluster roles and bindings, that are used by SQL Processors, if the deployment mode is set to KUBERNETES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.

To disable the creation of Kubernetes RBAC set: rbacEnabled: false

If you want to limit the permissions the Agent has against your Kubernetes cluster, you can use Role/RoleBinging resources instead. Follow this link in order to enable it.

If you are not using SQL Processors and want to limit permissions given to Agent's ServiceAccount, there are two options you can choose from:

rbacEnable: true - will enable the creation of ClusterRole and ClusterRoleBinding for service account mentioned above;
rbacEnable: true and namespaceScope: true - will enable creation of Role and RoleBinding which is more restrictive;

(Optional) Enable TLS connection with HQ

In this case, TLS has to be enabled on HQ. In case you haven't still enabled it, you can find details here to do it.

Enabling TLS in communication between HQ is being done in provisioning part of values.yaml.

In order to successfully enable TLS for the Agent you would need to:

additionalVolume & additionalVolumeMounts - with which you will mount truststore with CA certificate that HQ is using and which Agent will need to successfully pass the handshake.
additionalEnv - which will be used to securely read password to unlock truststore.
Enable ssl in provision.

values.yaml

# Additional Volume with CA that HQ uses
additionalVolumes:
  - name: hq-truststore
    secret:
      secretName: hq-agent-test-authority
additionalVolumeMounts:
  - name: hq-truststore
    mountPath: "/mnt/provision-secrets/hq"

lensesAgent:
 # Additional Env to read truststore password from secret
 additionalEnv:
    - name: LENSES_HQ_AGENT_TRUSTSTORE_PWD
      valueFrom:
        secretKeyRef:
          name: hq-agent-test-authority
          key: truststore.jks.password
 provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: [HQ_URL]
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
            sslEnabled:
              value: false
            sslTruststore:
              file: "/mnt/provision-secrets/gq/truststore.jks"
            sslTruststorePassword:
              value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

(Optional) Services

Enable a service resource in the values.yaml:

# Lenses service
service:
  enabled: true
  annotations: {}

(Optional) Controlling resources

To control the resources used by the Agent:

# Resource management
resources:
  requests:
    cpu: 1
    memory: 4Gi
  limits:
    cpu: 2
    memory: 5Gi

In case LENSES_HEAP_OPTS is not set explicitly it will be set implicitly.

Examples:

if no requests or limits are defined, LENSES_HEAP_OPTS will be set as -Xms1G -Xmx3G
If requests and limits are defined above defined values, LENSES_HEAP_OPTS will be set by formula -Xms[-Xmx / 2] -Xmx[limits.memory - 2]
If .Values.lenses.jvm.heapOpts it will override everything

Enabling SQL processors in K8s mode

To enable SQL processor in KUBERENTES mode and control the defaults:

lensesAgent:
  sql:
    # processorImage: eu.gcr.io/lenses-container-registry/lenses-sql-processor
    # processorImageTag: 2.3
    mode: KUBERNETES
    heap: 1024M
    minHeap: 128M
    memLimit: 1152M
    memRequest: 128M
    livenessInitialDelay: 60 seconds

To control the namespace Lenses can deploy processors, use the sql.namespaces value.

SQL Processor Role Binding

To achieve you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to.

For example:

Lenses namespace = lenses-ns
Processor namespace = lenses-proc-ns

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role
  namespace: lenses-proc-ns
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role-binding
  namespace: lenses-proc-ns
subjects:
- kind: ServiceAccount
  namespace: lenses-ns
  name: default
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io

Finally you need to define in the Agent configuration which namespaces the Agent has access to. Amend values.yaml to contain the following:

values.yaml

lensesAgent:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "lenses-processors"
        ]
      }

Prometheus metrics

Prometheus metrics are automatically exposed on port 9102 under /metrics.

lenses.conf

The main configurable options for lenses.conf are available in the values.yaml under the lenses object. These include:

Authentication
Database connections
SQL processor configurations

To apply other static configurations use lenses.append.conf, for example:

values.yaml

lensesAgent:
  append:
    conf: |
      lenses.interval.user.session.refresh=40000

Install the Chart

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio-preview https://lenses.jfrog.io/artifactory/api/helm/helm-charts-preview
helm repo update

Installing the Agent

Installing using cloned repository:

helm install lenses-agent charts/lenses-agent \
   --values charts/lenses-agent/values.yaml \
   --create-namespace --create lenses-agent

Installing using Helm repository:

terminal

helm install lenses-agent lensesio-preview/lenses-agent \
   --values values.yaml \
   --create-namespace --namespace lenses-agent \
   --version 6.0.0-alpha.5

Be aware that for the time being and for alpha purposes usage of --version is mandatory when deploying Helm chart through Helm repository.

Example Values files

Example of values.yaml

values.yaml

lensesAgent:
  storage:
    postgres:
      enabled: true
      host: [postgres.url]
      port: 5432
      username: postgres 
      password: changeMe
      database: agent
  hq:
    agentKey:
      secret:
        type: "createNew"
        name: "agentKey"
        value: "agent_key_*"
  provision:
    path: /mnt/provision-secrets
    connections:
      lensesHq:
        - name: lenses-hq
          version: 1
          tags: ['hq']
          configuration:
            server:
              value: hq-tls-test-lenses-hq.hq-agent-test.svc.cluster.local
            port:
              value: 10000
            agentKey:
              value: ${LENSESHQ_AGENT_KEY}
      kafka:
        # There can only be one Kafka cluster at a time
        - name: kafka
          version: 1
          tags: ['staging', 'pseudo-data-only']
          configuration:
            kafkaBootstrapServers:
              value:
                - PLAINTEXT://kafka-1.svc.cluster.local:9092
                - PLAINTEXT://kafka-2.svc.cluster.local:9092
            protocol:
              value: PLAINTEXT
            # Metrics are strongly suggested for better Kafka cluster observability
            metricsType:
              value: JMX
            metricsPort:
              value: 9581

You can also find examples in the Helm chart repo.

Docker

This page describes installing Lenses with Docker Image.

Deploying HQ

This page describes deploying Lenses HQ via docker.

The HQ docker image can be configured via volume mounts for the configuration file.

The HQ looks for the config.yaml in the current working directory. This is the root directory for Docker.

Running the Docker

terminal

docker run --name lenses-hq \
--network panoptes \
-p 8080:8080 \
-v $(pwd)/config.yaml:/config.yaml\
lensting/lenses-hq:6-preview

Prerequisites

The main pre-requirements that has to be fulfilled before Lenses HQ container can be started and those are:

Postgres database

Complete configuration file

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

Main configuration file that has to be configured before running docker command is config.yaml.

Sample configuration file is following:

config.yaml

auth:
  administrators:
    - admin
    - admin@company.com
  users:
    - username: admin
      password: $2a$20$6kxYW6cAOpc3lxzne5dHKuZOPR.dMxtQvvE9scw7.dreKx27AR0L2
  sessionDuration: 24h
  saml:
    enabled: true
    baseURL: https://lenses6.company.com
    entityID: https://lenses6.company.com
    metadata: |          
         <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor
          xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"
            </md:IDPSSODescriptor>
          </md:EntityDescriptor>
          
    userCreationMode: manual
    groupMembershipMode: manual
    uiRootURL: /
    groupAttributeKey: groups
    authnRequestSignature:
      enabled: false
http:
  address: :8080
  accessControlAllowOrigin:
    - https://lenses6.company.com
  accessControlAllowCredentials: false
  secureSessionCookies: true
  
agents:
  address: :10000
  
database:
  host: postgres-postgresql.postgres.svc.cluster.local:5432
  username: $(LENSESHQ_PG_USERNAME)
  password: $(LENSESHQ_PG_PASSWORD)
  schema:
  database: lenseshq
  TLS: false
license:
  key: license_key_
  acceptEULA: true
logger:
  mode: text
  level: debug
metrics:
  prometheusAddress: :9090

More about configuration options you can find on the HQ configuration page.

Deploying an Agent

This page describes deploying an Lenses Agent via Docker.

The Agent docker image can be configured via environment variables or via volume mounts for the configuration files.

Running the Docker

Please check "File configuration" before running the command

terminal

docker run --name lenses-agent \
-v $(pwd)/provisioning.yaml:/mnt/provision-secrets/provisioning.yaml \
-v $(pwd)/lenses.conf:/data/lenses.conf \
-e LENSES_PROVISIONING_PATH=/mnt/provision-secrets \
lensting/lenses-agent:6-preview

File configuration

In order for command above provisioning yaml has to be configured.

There are two mandatory connections:

HQ which requires AgentKey (LENSESHQ_AGENT_KEY) - this key is being created once user registers "New environment" in HQ;
Kafka connection

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: ${LENSESHQ_AGENT_KEY}
    port:
      value: 10000
    server:
      value: hq
  name: lenses-hq
  tags:
  - hq
  version: 1

kafka:
- configuration:
   kafkaBootstrapServers:
      value:
      - PLAINTEXT://kafka:9092
   protocol:
      value: PLAINTEXT
  name: kafka
  tags:
  - demo
  version: 1

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses.conf can be mounted directly as

/mnt/settings/lenses.conf

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

/data/storage
/data/plugins
/data/logs
/data/kafka-streams-state

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Agent upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Agent may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Agent TLS and Global JVM Trust Store

By default, the the serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets
Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

File / Variable Name

Description

FILECONTENT_JVM_SSL_TRUSTSTORE

The SSL/TLS trust store to use as the global JVM trust store. Add to LENSES_OPTS the property javax.net.ssl.trustStore

FILECONTENT_JVM_SSL_TRUSTSTORE_PASSWORD

Τhe trust store password. If set, the startup script will add automatically to LENSESOPTS the property javax.net.ssl.trustStorePassword (**_base64 not supported**)

FILECONTENT_LENSES_SSL_KEYSTORE

The SSL/TLS keystore to use for the TLS listener for the Agent

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting the Agent.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.

Linux

This page describes install the Lenses via a Linux archive

Deploying HQ

This page describes the install of the Lenses Agent via an archive on Linux.

To install the HQ from the archive you must:

Extract the archive
Configure the HQ
Start the HQ

Extracting the archive

Installation link

Link to archives can be found here: https://archive.lenses.io/pub/testing/lenses/

Extract the archive using the following command

terminal

tar -xvf lenses-hq-6.0.0-alpha.20-linux-amd64.tar.gz -C lenses-hq

Inside the extract archive, you will find.

terminal

   lenses-hq
   ├── lenses-hq-6.0.0-alpha.20-linux-amd64

Configuring the HQ

In order to properly configure HQ, two core components are necessary:

Postgres database

Configure Authentication

To set up authentication, there are multiple methods available.

You can choose between:

password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.

Both password based and SAML / SSO authentication methods can be used alongside each other.

First to cover is users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. The passwords are hashed using bcrypt for security purposes, ensuring that they are stored securely.

Second to cover will be administrators. It serves as definition of user emails which will have highest level of permissions upon authentication to HQ.

config.yaml

auth:
  users:
    - username: admin
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple").
  administrators:
    - admin
    - admin@example.com
  saml:
    enabled: true
    metadata: |-
      <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor>
      ...
      ...
      </md:EntityDescriptor>
    baseURL: https://example.com
    entityID: https://example.com
    userCreationMode: sso
    groupMembershipMode: sso

Configure HTTP endpoint

Another part which has to be set in order to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around HTTP endpoint of the HQ itself and how users will interact with.

Definition of HTTP object is as follows:

config.yaml

http:
  address: :8080
  accessControlAllowOrigin:
    - https://example.com
  accessControlAllowCredentials: false
  secureSessionCookies: true
  tls:
    enabled: true
    cert: ""
    key: ""

More about setting up TLS can be read here.

Configure Agent endpoint

After correctly configuring authentication strategy and connection endpoint , agent handling is the last most important box to tick.

The Agent's object is defined as follows:

config.yaml

agents:
  address: :10000
  tls:
    enabled: true
    cert: ""
    key: ""

More about setting up TLS can be read here.

Configure database

Prerequisite:

Running Postgres instance;
Created database for HQ;
Username (and password) which has access to created database;

In order to successfully run HQ, storage within config.yaml has to be defined first.

Definition of storage object is as follows:

config.yaml

database:
  host: postgres:5432
  username: panoptes
  password: password
  database: panoptes
  params:
    sslmode: require

Configure license and accept EULA

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

config.yaml

license:
  key: license_key_*
  acceptEULA: true

Final Configuration File

If you have meticulously followed all the outlined steps, your config.yaml file should mirror the example provided below, fully configured and ready for deployment. This ensures your system is set up correctly with all necessary settings for authentication, database connection, and other configurations optimally defined.

config.yaml

auth:
  users:
    - username: admin
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple").
  administrators:
    - admin
    - admin@example.com
  saml:
    enabled: true
    metadata: |-
      <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor>
      ...
      ...
      </md:EntityDescriptor>
    baseURL: https://example.com
    entityID: https://example.com
    userCreationMode: sso
    groupMembershipMode: sso
http:
  address: ":8080"
  accessControlAllowOrigin:
    - https://example.com
agents:
  address: ":10000"
database:
  host: postgres:5432
  username: panoptes
  password: password
  database: panoptes
  params:
    sslmode: require
licence:
  key: license_key_*
logger:
  mode: text
  level: debug

Starting the HQ

Start Lenses by running:

terminal

./backend-darwin-amd64

or pass the location of the config file:

terminal

./backend-darwin-amd64 config.yaml

If you do not pass the location of the config file, the HQ will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

Once HQ starts, it will be listening on the https://localhost:8080

To stop HQ, press CTRL+C.

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

[Unit]
Description=Run HQ service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses-hq
ExecStart=/opt/lenses-hq /etc/lenses-hq/config.yaml

[Install]
WantedBy=multi-user.target

Deploying an Agent

This page describes the install of the Lenses Agent via an archive on Linux.

To install the Agent from the archive you must:

Extract the archive
Configure the Agent
Start the Agent

Extracting the archive

Installation link

Link to archives can be found here: https://archive.lenses.io/pub/testing/lenses/

Extract the archive using the following command

terminal

tar -xvf lenses.tar.gz -C lenses

Inside the extract archive, you will find.

   lenses
   ├── lenses.conf       ← edited and renamed from .sample
   ├── logback.xml
   ├── logback-debug.xml
   ├── bin/
   ├── lib/
   ├── licences/
   ├── logs/             ← created when you run Lenses
   ├── plugins/
   ├── storage/          ← created when you run Lenses
   └── ui/

Configure the Agent

To configure the agents connection to Postgres and its provisioning file. See here in the quickstart.

Once the agent files are configure you can continue to start the agent.

The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.

Provisioning

Configure HQ

To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: [LENSES_HQ_URL]
      port:
        value: 10000
      agentKey:
        value: ${LENSESHQ_AGENT_KEY}
      sslEnabled:
        value: true
      sslTruststore:
        file: "/mnt/provision-secrets/hq/truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

Agent key reference

Agent key within provisioning.yaml can be referenced as a:

environment variable shown in example above
inline string

Configure Kafka

There are many Kafka flavours today in the market. Good news is that Lenses support all flavours of Kafka and we are trying hard to keep documention up to date.

In the following link you can find provisioning examples for the most common Kafka flavours.

Other connections

There are also provisioning examples for other components:

Schema Registry
Kafka Connect
Zookeeper
AWS
Alerts & Audit integrations

Starting the Agent

Provisioning file path

If you configured provisioning.yaml make sure to place following property:

lenses.conf

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir

Start Lenses by running:

terminal

bin/lenses

or pass the location of the config file:

terminal

bin/lenses lenses.conf

If you do not pass the location of lenses.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the lenses.conf to be readable only by the lenses user.

chmod 0600 /path/to/lenses.conf
chown [lenses-user]:root /path/to/lenses.conf

The agent needs write access in 4-5 places in total:

[RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:
1. [RUNTIME DIRECTORY]/logs Where logs are stored
2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.
3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.
4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.
5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

Back-up this location for disaster recovery

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

Mount /tmp without noexec
or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

LENSES_OPTS="-Dorg.xerial.snappy.tempdir=/path/to/exec/tmp -Djava.io.tmpdir=/path/to/exec/tmp"

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

[Unit]
Description=Run Agent service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback.xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

export LENSES_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
bin/lenses

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 8GB RAM /4 CPUs and 20GB disk space.

Configuration

This page describes how to configure Lenses.

Authentication

This page describes the authentication methods supported in Lenses.

Authentication is configured in HQ.

Users can authentication is two ways. Basic authentication and SSO / SAML. Additional specific users can be assigned as admin accounts.

Admin Account

This page describes how to configure admin accounts in Lenses.

You can configure a list of the principals (users, service accounts) that have root admin access. Access control allows any API operation performed by such principals. If not set, it will default to [].

Admin accounts are set in the config.yaml for HQ under the http.adminstrators key, as an array of usernames.

config.yaml

http:
  administrators:
  - brian.lifeof@lenses.io
  - bob.builder@lenses.io

Basic Authentication

This page describes configuring basic authentication in Lenses.

Basic authentication is set in the config.yaml for HQ under the http.users key, as an array of usernames and passwords.

To enhance security, it's essential that passwords in the config.yaml file are stored in bcrypt format.

This ensures that the passwords are hashed and secure rather than stored in plaintext. For instance, instead of using "builder" directly, it should be hashed using bcrypt.

An example of a bcrypt-hashed password looks like this: $2a$12$XQW..XQrtZXCvbQWertqQeFi/1KoQW4eNephNXTfHqtoW9Q4qih5G.

Always ensure that you replace plaintext passwords with their bcrypt counterparts to securely authenticate users.

config.yaml

auth:
  users:
  - username: bob
    password: $2a$12$XQW..XQrtZXCvbQWertqQeFi/1KoQW4eNephNXTfHqtoW9Q4qih5G 
  - username: brain
    password: $2a$12$XQW..XQrtZXCvbQWertqQeFi/1KoQW4eNephNXTfHqtoW9Q4qih5G

SSO & SAML

This page describes configure SSO & SAML in Lenses for authentication.

Overview

This page gives an overview of SSO & SAML for authentication with Lenses.

Users

Control of how user create with SSO is determined by the SSO User Creation Mode. There are two modes:

Manual
SSO

With manual mode, only users that pre-created in HQ can login.

With sso mode, users that do not already exists are created and logged in.

Group Mapping

Control of how a user's group membership should be handled in relation to SSO is determined by the SSO Group Membership Mode. There are two modes:

Manual
SSO

With the manual mode, the information about the group membership returned from an Identity Provider will not be used and a user will only be a member of groups that were explicitly assigned to them in HQ.

With the sso mode, group information from Identity Provider (IdP) will be used. On login, a user's group membership is set to the groups listed in the IdP.

Groups that do not exist in HQ are ignored.

SAML configuration is defined in the config.yaml provided to HQ. For more information on the configuration options see here.

config.yaml

http:
  saml:
    metadata: |-

The follow SSO / SAML providers are supported.

Azure SSO

This page describes configuring Azure SSO for Lenses authentication.

Learn more here about Azure SSO

Add Lenses from Azure SSO gallery

Go to Enterprise applications->New Application.

Search for Lenses.io in the gallery directory.

Choose a name for Lenses e.g. Lenses.io and click Add.

Enable SSO

On the overview page select Single Sign On.

Configure SAML

Remember to activate HTTPS on HQ. See TLS.

Setting

Value

Identifier (Entity ID)

Use the base url of the Lenses installation e.g. https://lenses-dev.example.com

Reply URL

Use the base url with the callback details e.g. https://lenses-dev.example.com/api/v2/auth/saml/callback?client_name=SAML2Client

Sign on URL

Use the base url

Download SAML Certificates

Download the Federation Metadata XML.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

Google SSO

This page describes configuring Google SSO for Lenses authentication.

Create a custom attribute for Lenses groups

Google doesn't expose the groups, or organization unit, of a user to a SAML app. This means we must set up a custom attribute for the Lenses groups that each user belongs to.

Open the Google Admin console from an administrator account.

Click the Users button
Select the More dropdown and choose Manage custom attributes
Click the Add custom attribute button
Fill the form to add a Text, Multi-value field for Lenses Groups, then click Add

Learn more about Google custom attributes

Assign Lenses groups attributes to Google users

The attribute values should correspond exactly with the names of groups created within Lenses.

Open the Google Admin console from an administrator account.

Click the Users button
Select the user to update
Click User information
Click the Lenses Groups attribute
Enter one or more groups and click Save

Add Google custom SAML app

Learn more about Google custom SAML apps

Open the Google Admin console from an administrator account.
Click the Apps button
Click the SAML apps button
Select the Add App dropdown and choose Add custom SAML app
Run through the below steps

App Details

Enter a descriptive name for the Lenses installation
Upload a Lenses icon

This will appear in the Google apps menu once the app is enabled

Configure SAML

Service provider details

Given the base URL of the Lenses installation, e.g. https://lenses-dev.example.com, fill out the settings:

Setting

Value

ACS URL

Use the base url with the callback path e.g. https://lenses-dev.example.com/api/v2/auth/saml/callback?client_name=SAML2Client

Entity ID

Use the base url e.g. https://lenses-dev.example.com

Start URL

Leave empty

Signed Response

Leave unchecked

Name ID format

Leave as UNSPECIFIED

Name ID

Leave as Basic Information > Primary Email

Attribute mapping

Add a mapping from the custom attribute for Lenses groups to the app attribute groups

Enable the app

From the newly added app details screen, select User access
Turn on the service

Lenses will reject any user that doesn't have the groups attribute set, so enabling the app for all users in the account is a good option to simplify ongoing administration.

Download the Federation Metadata XML file with the Google IdP details.

Download SAML Certificates

Click Download Metadata and save the metadata file for configuring Lenses.Configure SAML in HQ.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

Keycloak SSO

This page describes configuring Keycloak SSO for Lenses authentication.

Create a new SAML application client in Keycloak

Go to Clients
Click Create
Fill in the details: see the table below.
Click Save

Setting

Value

Client ID

Use the base.url of the Lenses installation e.g. https://lenses-dev.example.com

Client Protocol

Set it to saml

Client Saml Endpoint

This is the Lenses API point for Keycloak to call back. Set it to [BASE_URL]/api/v2/auth/saml/callback?client_name=SAML2Client. e.g. https://lenses-dev.example.com/api/v2/auth/saml/callback?client_name=SAML2Client

Update Client Settings

Change the settings on client you just created to:

Setting

Value

Name

Lenses

Description

(Optional) Add a description to your app.

SAML Signature Name

KEY_ID

Client Signature Required

OFF

Force POST Binding

Front Channel Logout

OFF

Force Name ID Format

Name ID Format

Root URL

Use the base.url of the Lenses installation e.g. https://lenses-dev.example.com

Valid Redirect URIs

Use the base.url of the Lenses installation e.g. https://lenses-dev.example.com

Map users to groups

Configure Keycloak to communicate groups to Lenses. Head to the Mappers (under Client scope tab) section.

Click Create
Fill in the details: see table below.
Click Save

Setting

Value

Name

Groups

Mapper Type

Group list

Group attribute name

groups (case-sensitive)

Single Group Attribute

Full group path

OFF

Download SAML Certificates

Download the Federation Metadata XML file with the Keycloak IdP details.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

Okta SSO

This page describes configuring Okta SSO for Lenses authentication.

Set up Okta IdP

Lenses is available directly in Okta’s Application catalog.

Add application in the Catalog

Go to Applications->Applications
Click Add Application
Search for Lenses
Select by pressing Add

Set General Settings

App label: Lenses
Set the base url of your lenses installation e.g. https://lenses-dev.example.com
Click Done

Download SAML Certificates

Download the Federation Metadata XML file with the OktaIdP details.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

OneLogin SSO

This page describes configuring OneLogin SSO for Lenses authentication.

Set up OneLogin Id

Lenses is available in the OneLogin Application catalog.

Visit OneLogin’s Administration console.

Select Applications->Applications->Add App
Search and select Lenses
Optionally add a description and click save

Add Lenses via the Application Catalog

In the Configuration section set the base path from the url of the Lenses installation e.g. lenses-dev.example.com (without the https://)
Click Save

Download SAML Certificates

Download the Federation Metadata XML file with the OneLogin IdP details.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

Generic SSO

This page describes configuring a Generic SSO provider for Lenses authentication.

Configure SAML in HQ

SAML configuration is set in HQ's config.yaml file. See here for more details.

HQ

This page describe the Lenses Agent configuration.

HQ's configuration is defined in the config.yaml file

EULA

To accept the Lenses EULA, set the following in the lenses.conf file:

Without accepting the EULA the Agent will not start! See License.

It has the following top level groups:

Name

Required

Default

Type

Description

http

Yes

n/a

Configures everything involving the HTTP.

agents

Yes

n/a

Controls the agent handling.

database

Yes

n/a

Configures database settings.

logger

Yes

n/a

Sets the logger behaviour.

metrics

Yes

n/a

Controls the metrics settings.

license

Yes

n/a

Holds the license key.

auth

Yes

n/a

Configures authentication and authorisation

AuthConfig

Configures authentication and authorisation.

It has the following fields:

Name

Required

Default

Type

Description

administrators

[]

strings

Grants root access to principals.

saml

n/a

Contains SAML2 IdP configuration.

users

[]

Array

Creates initial users for password based authentication.

AuthConfig: administrators

Lists the names of the principals (users, service accounts) that have root access. Access control allows any API operation performed by such principals. Optional. If not set, it will default to [].

AuthConfig: saml

Contains SAML2 IdP configuration. Please refer here for its structure.

HTTPConfig

Configures everything involving the HTTP.

It has the following fields:

Name

Required

Default

Type

Description

address

Yes

n/a

string

Sets the address the HTTP server listens at.

accessControlAllowOrigin

["*"]

strings

Sets the value of the "Access-Control-Allow-Origin" header.

accessControlAllowCredentials

false

boolean

Sets the value of the "Access-Control-Allow-Credentials" header.

secureSessionCookies

true

boolean

Sets the "Secure" attribute on session cookies.

tls

Yes

n/a

Contains TLS configuration.

HTTPConfig: address

Sets the address the HTTP server listens at.

Example value: 127.0.0.1:80.

HTTPConfig: accessControlAllowOrigin

Sets the value of the "Access-Control-Allow-Origin" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to ["*"].

HTTPConfig: accessControlAllowCredentials

Sets the value of the "Access-Control-Allow-Credentials" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to false.

HTTPConfig: secureSessionCookies

Sets the "Secure" attribute on authentication session cookies. When set, a browser sends such cookies not over unsecured HTTP (expect for localhost). If running Lenses HQ over unsecured HTTP, set this to false. Optional. If not set, it will default to true.

HTTPConfig: tls

Contains TLS configuration. Please refer here for its structure.

SAMLConfig

Contains SAML2 IdP configuration.

It has the following fields:

Name

Required

Default

Type

Description

metadata

Yes

n/a

string

Contains the IdP issued XML metadata blob.

baseURL

Yes

n/a

string

Defines base URL of HQ for IdP redirects.

uiRootURL

/

string

Controls where to redirect to upon successful authentication.

entityID

Yes

n/a

string

Defines the Entity ID.

groupAttributeKey

groups

string

Sets the attribute name for group names.

userCreationMode

manual

string

Controls how the creation of users should be handled in relation to SSO information.

groupMembershipMode

manual

string

Controls how the management of a user's group membership should be handled in relation to SSO information.

SAMLConfig: metadata

Contains the IdP issued XML metadata blob.

Example value: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>.

SAMLConfig: baseURL

Defines the base URL of Lenses HQ; the IdP redirects back to here on success.

Example value: https://hq.example.com.

SAMLConfig: uiRootURL

Controls where the backend redirects to after having received a valid SAML2 assertion. Optional. If not set, it will default to /.

Example value: /.

SAMLConfig: entityID

Defines the Entity ID.

Example value: https://hq.example.com.

SAMLConfig: groupAttributeKey

Sets the attribute name from which group names are extracted in the SAML2 assertions. Different providers use different names. Okta, Keycloak and Google use "groups". OneLogin uses "roles". Azure uses "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups". Optional. If not set, it will default to groups.

Example value: groups.

SAMLConfig: userCreationMode

Controls how the creation of users should be handled in relation to SSO information. With the 'manual' mode, only users that currently exist in HQ can login. Users that do not exist are rejected. With the 'sso' mode, users that do not exist are automatically created. Allowed values are manual or sso. Optional. If not set, it will default to manual.

SAMLConfig: groupMembershipMode

Controls how the management of a user's group membership should be handled in relation to SSO information. With the 'manual' mode, the information about the group membership returned from an Identity Provider will not be used and a user will only be a member of groups that were explicitly assigned to him locally. With the 'sso' mode, group information from Identity Provider (IdP) will be used. On login, a user's group membership is set to the groups listed in the IdP. Groups that do not exist in HQ are ignored. Allowed values are manual or sso. Optional. If not set, it will default to manual.

AgentsConfig

Controls the agent handling.

It has the following fields:

Name

Required

Default

Type

Description

address

Yes

n/a

string

Sets the address the agent server listens at.

tls

Yes

n/a

Contains TLS configuration.

AgentsConfig: address

Sets the address the agent server listens at.

Example value: 127.0.0.1:3000.

AgentsConfig: tls

Contains TLS configuration. Please refer here for its structure.

TLSConfig

Contains TLS configuration.

It has the following fields:

Name

Required

Default

Type

Description

enabled

Yes

n/a

boolean

Enables or disables TLS.

cert

string

Sets the PEM formatted public certificate.

key

string

Sets the PEM formatted private key.

verboseLogs

false

boolean

Enables verbose TLS logging.

TLSConfig: enabled

Enables or disables TLS.

Example value: false.

TLSConfig: cert

Sets the PEM formatted public certificate. Optional. If not set, it will default to ``.

Example value: -----BEGIN CERTIFICATE----- EXampLeRanDoM ... -----END CERTIFICATE----- .

TLSConfig: key

Sets the PEM formatted private key. Optional. If not set, it will default to ``.

Example value: -----BEGIN PRIVATE KEY----- ExAmPlErAnDoM ... -----END PRIVATE KEY----- .

TLSConfig: verboseLogs

Enables additional logging of TLS settings and events at debug level. The information presented might be a bit too much for day to day use but can provide extra information for troubleshooting TLS configuration. Optional. If not set, it will default to false.

DatabaseConfig

Configures database settings.

It has the following fields:

Name

Required

Default

Type

Description

host

Yes

n/a

string

Sets the name of the host to connect to.

username

string

Sets the username to authenticate as.

password

string

Sets the password to authenticate as.

database

Yes

n/a

string

Sets the database to use.

schema

string

Sets the schema to use.

TLS

false

boolean

Enables TLS.

params

{}

DBConnectionParams

Provides fine-grained control.

DatabaseConfig: host

Sets the name of the host to connect to. A comma-separated list of host names is also accepted; each host name in the list is tried in order.

Example value: postgres:5432.

DatabaseConfig: username

Sets the username to authenticate as. Optional. If not set, it will default to ``.

Example value: johhnybingo.

DatabaseConfig: password

Sets the password to authenticate as. Optional. If not set, it will default to ``.

Example value: my-password.

DatabaseConfig: database

Sets the database to use.

Example value: my-database.

DatabaseConfig: schema

Sets the schema to use. Optional. If not set, it will default to ``.

Example value: my-schema.

DatabaseConfig: TLS

Enables TLS. In PostgreSQL connection string terms, setting TLS to false corresponds to sslmode=disable; setting TLS to true corresponds to sslmode=verify-full. For more fine-grained control, specify sslmode in the params which takes precedence. Optional. If not set, it will default to false.

Example value: true.

DatabaseConfig: params

Contains connection string parameters as key/values pairs. It allow fine-grained control of connection settings. The parameters can be found here: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS Optional. If not set, it will default to {}.

Example value: {"application_name":"example"}.

LoggerConfig

Sets the logger behaviour.

It has the following fields:

Name

Required

Default

Type

Description

mode

Yes

n/a

string

Controls the format of the logger's output.

level

info

string

Controls the level of the logger.

LoggerConfig: mode

Controls the format of the logger's output. Allowed values are text or json.

LoggerConfig: level

Controls the level of the logger. Allowed values are info or debug. Optional. If not set, it will default to info.

MetricsConfig

Controls the metrics settings.

It has the following fields:

Name

Required

Default

Type

Description

prometheusAddress

:9090

string

Sets the Prometheus address.

MetricsConfig: prometheusAddress

Sets the address at which Prometheus metrics are served. Optional. If not set, it will default to :9090.

License

Holds the license key.

It has the following fields:

Name

Required

Default

Type

Description

key

Yes

n/a

string

Sets the license key.

acceptEULA

Yes

fals

boolean

License: key

Sets the license key. An HQ key starts with "licensekey".

License: acceptEULA

Accepts the Lenses EULA.

Configuration Reference

HQ's configuration is defined in the config.yaml file

EULA

To accept the Lenses EULA, set the following in the lenses.conf file:

Without accepting the EULA the Agent will not start! See License.

It has the following top level groups:

Name

Required

Default

Type

Description

http

Yes

n/a

Configures everything involving the HTTP.

agents

Yes

n/a

Controls the agent handling.

database

Yes

n/a

Configures database settings.

logger

Yes

n/a

Sets the logger behaviour.

metrics

Yes

n/a

Controls the metrics settings.

license

Yes

n/a

Holds the license key.

auth

Yes

n/a

Configures authentication and authorisation

AuthConfig

Configures authentication and authorisation.

It has the following fields:

Name

Required

Default

Type

Description

administrators

[]

strings

Grants root access to principals.

saml

n/a

Contains SAML2 IdP configuration.

users

[]

Array

Creates initial users for password based authentication.

AuthConfig: administrators

Lists the names of the principals (users, service accounts) that have root access. Access control allows any API operation performed by such principals. Optional. If not set, it will default to [].

AuthConfig: saml

Contains SAML2 IdP configuration. Please refer here for its structure.

HTTPConfig

Configures everything involving the HTTP.

It has the following fields:

Name

Required

Default

Type

Description

address

Yes

n/a

string

Sets the address the HTTP server listens at.

accessControlAllowOrigin

["*"]

strings

Sets the value of the "Access-Control-Allow-Origin" header.

accessControlAllowCredentials

false

boolean

Sets the value of the "Access-Control-Allow-Credentials" header.

secureSessionCookies

true

boolean

Sets the "Secure" attribute on session cookies.

tls

Yes

n/a

Contains TLS configuration.

HTTPConfig: address

Sets the address the HTTP server listens at.

Example value: 127.0.0.1:80.

HTTPConfig: accessControlAllowOrigin

Sets the value of the "Access-Control-Allow-Origin" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to ["*"].

HTTPConfig: accessControlAllowCredentials

Sets the value of the "Access-Control-Allow-Credentials" header. This is only relevant when serving the backend from a different origin than the UI. Optional. If not set, it will default to false.

HTTPConfig: secureSessionCookies

HTTPConfig: tls

Contains TLS configuration. Please refer here for its structure.

SAMLConfig

Contains SAML2 IdP configuration.

It has the following fields:

Name

Required

Default

Type

Description

metadata

Yes

n/a

string

Contains the IdP issued XML metadata blob.

baseURL

Yes

n/a

string

Defines base URL of HQ for IdP redirects.

uiRootURL

/

string

Controls where to redirect to upon successful authentication.

entityID

Yes

n/a

string

Defines the Entity ID.

groupAttributeKey

groups

string

Sets the attribute name for group names.

userCreationMode

manual

string

Controls how the creation of users should be handled in relation to SSO information.

groupMembershipMode

manual

string

Controls how the management of a user's group membership should be handled in relation to SSO information.

SAMLConfig: metadata

Contains the IdP issued XML metadata blob.

Example value: <?xml version="1.0" ... (big blob of xml) </md:EntityDescriptor>.

SAMLConfig: baseURL

Defines the base URL of Lenses HQ; the IdP redirects back to here on success.

Example value: https://hq.example.com.

SAMLConfig: uiRootURL

Controls where the backend redirects to after having received a valid SAML2 assertion. Optional. If not set, it will default to /.

Example value: /.

SAMLConfig: entityID

Defines the Entity ID.

Example value: https://hq.example.com.

SAMLConfig: groupAttributeKey

Example value: groups.

SAMLConfig: userCreationMode

SAMLConfig: groupMembershipMode

AgentsConfig

Controls the agent handling.

It has the following fields:

Name

Required

Default

Type

Description

address

Yes

n/a

string

Sets the address the agent server listens at.

tls

Yes

n/a

Contains TLS configuration.

AgentsConfig: address

Sets the address the agent server listens at.

Example value: 127.0.0.1:3000.

AgentsConfig: tls

Contains TLS configuration. Please refer here for its structure.

TLSConfig

Contains TLS configuration.

It has the following fields:

Name

Required

Default

Type

Description

enabled

Yes

n/a

boolean

Enables or disables TLS.

cert

string

Sets the PEM formatted public certificate.

key

string

Sets the PEM formatted private key.

verboseLogs

false

boolean

Enables verbose TLS logging.

TLSConfig: enabled

Enables or disables TLS.

Example value: false.

TLSConfig: cert

Sets the PEM formatted public certificate. Optional. If not set, it will default to ``.

Example value: -----BEGIN CERTIFICATE----- EXampLeRanDoM ... -----END CERTIFICATE----- .

TLSConfig: key

Sets the PEM formatted private key. Optional. If not set, it will default to ``.

Example value: -----BEGIN PRIVATE KEY----- ExAmPlErAnDoM ... -----END PRIVATE KEY----- .

TLSConfig: verboseLogs

DatabaseConfig

Configures database settings.

It has the following fields:

Name

Required

Default

Type

Description

host

Yes

n/a

string

Sets the name of the host to connect to.

username

string

Sets the username to authenticate as.

password

string

Sets the password to authenticate as.

database

Yes

n/a

string

Sets the database to use.

schema

string

Sets the schema to use.

TLS

false

boolean

Enables TLS.

params

{}

DBConnectionParams

Provides fine-grained control.

DatabaseConfig: host

Sets the name of the host to connect to. A comma-separated list of host names is also accepted; each host name in the list is tried in order.

Example value: postgres:5432.

DatabaseConfig: username

Sets the username to authenticate as. Optional. If not set, it will default to ``.

Example value: johhnybingo.

DatabaseConfig: password

Sets the password to authenticate as. Optional. If not set, it will default to ``.

Example value: my-password.

DatabaseConfig: database

Sets the database to use.

Example value: my-database.

DatabaseConfig: schema

Sets the schema to use. Optional. If not set, it will default to ``.

Example value: my-schema.

DatabaseConfig: TLS

Example value: true.

DatabaseConfig: params

Example value: {"application_name":"example"}.

LoggerConfig

Sets the logger behaviour.

It has the following fields:

Name

Required

Default

Type

Description

mode

Yes

n/a

string

Controls the format of the logger's output.

level

info

string

Controls the level of the logger.

LoggerConfig: mode

Controls the format of the logger's output. Allowed values are text or json.

LoggerConfig: level

Controls the level of the logger. Allowed values are info or debug. Optional. If not set, it will default to info.

MetricsConfig

Controls the metrics settings.

It has the following fields:

Name

Required

Default

Type

Description

prometheusAddress

:9090

string

Sets the Prometheus address.

MetricsConfig: prometheusAddress

Sets the address at which Prometheus metrics are served. Optional. If not set, it will default to :9090.

License

Holds the license key.

It has the following fields:

Name

Required

Default

Type

Description

key

Yes

n/a

string

Sets the license key.

acceptEULA

Yes

fals

boolean

License: key

Sets the license key. An HQ key starts with "licensekey".

License: acceptEULA

Accepts the Lenses EULA.

API reference

Users

Groups

Roles

Service Accounts

Environments

License

Agent

This page describe the Lenses Agent configuration.

Overview

This page describes an overview of the Lenses Agent configuration.

The Agent configuration is driven by two files:

lenses.conf
provisioning.yaml

lenses.conf holds all the database connections and low level options for the agent.

provisioning.yaml holds the your Kafka cluster and supporting services, that the Agent is to connect to. In addition it defines the connection to HQ. The provisioning.yaml is watched by the Agent, so any changes made, if valid, are applied. See for more information. Without provisioning your agent can not connect to HQ.

Provisioning

This page describes how to setup connections to Kafka and other services and have changes applied automatically for the Lenses Agent.

Overview

This page describes an overview of Lenses Agent Provisioning.

As of version 6.0 the calling the Rest endpoint for provisioning is no longer available.

Connections are defined in the provisioning.yaml file. The Agent will watch the file and resolve the desired state, applying connections defined in the file.

Defining a Connection

Connections are defined in the provisioning.yaml. This file is divided into components, each component representing a type of connection.

Each component is mandatory:

Name - This is the free name of the connection
Version set to 1
Configuration - This is a list of keys/values dependent on the component type.

Example provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm, the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and resolve the value at runtime.

sslKeystorePassword:
  value: ${ENV_VAR_NAME}

Referencing files

Many connections need files, for example, to secure Kafka with SSL you will need a key store and optionally a trust store.

To reference a file in the provisioning.yaml, for example, given:

    configuration:
      protocol:
        value: SASL_SSL
      sslKeystore:
        file: "my-keystore.jks"

a file called my-keystore.jks is expected in the same directory.

HQ

This page describes connection a Lenses Agent with HQ

To be able to view and drill into your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: hq-tls-test-lenses-hq.hq-agent-test.svc.cluster.local
      port:
        value: 10000
      agentKey:
        value: ${LENSESHQ_AGENT_KEY}
      sslEnabled:
        value: true
      sslTruststore:
        file: "/mnt/provision-secrets/grpc/truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

Kafka

This page describes how to connect the Lenses Agent to your Kafka brokers.

The Lenses Agent can connect to any Kafka cluster or service exposing the Apache Kafka APIs and supporting the authentication methods offered by Apache Kafka.

Apache Kafka

This page describes connecting the Lenses Agent to Apache Kafka.

A Kafka connection is required for the agent to start. You can connect to Kafka via:

Plaintext (no credentials an unencrypted)
SSL (no credentials an encrypted)
SASL Plaintext and SASL SSL

Plaintext

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.
protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of the Agent does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword

SASL Plaintext vs SASL SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is:

The transport layer is encyrpted (SSL)
The SASL mechanism for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

To use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

When encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

SASL SSL

Mechanism PLAIN

Encrypted communication and basic username and password for authentication.

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";

Mechanism GSSAPI

In order to use Kerberos authentication, a Kerberos Connection should be created beforehand.

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword  
    saslMechanism: 
      value: GSSAPI
    saslJaasConfig:
      value: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";      
     keytab:
       file: /path/to/keytab.jks

SASL Plaintext

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

Mechanism SCRAM-SHA-256

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

Mechanism SCRAM-SHA-512

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-512
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

Aiven

This page describes configuring Lenses to connect to Aiven.

Find your Service URI

From the Aiven, locate your Service URI and set it as the bootstrap servers.

Configure Provisioning

Set the following in the provisioning.yaml, replacing Service URI, username and password from your Aiven account.

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[Service URI]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="[your-username]"
        password="[your-password]";

AWS MSK

This page describes connection the Lenses Agent to a AWS MSK cluster.

It is recommended to install the Agent on an EC2 instance or with EKS in the same VPC as your MSK cluster. The Agent can be installed and preconfigured via the AWS Marketplace.

Open network connectivity

Edit the AWS MSK security group in the AWS Console and add the IP address of your Agent installation.

Enable Open Monitoring

If you want to have the Agent collect JMX metrics you have to enable Open Monitoring on your MSK cluster. Follow the AWS guide here.

Select your MSK endpoint

Depending on your MSK cluster, select the endpoint and protocol you want to connect with.

It is not recommended to use Plaintext for secure environments. For these environments use TLS or IAM.

When the Agent is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

Configure Provisioning

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

AWS MSK Serverless

This page describes how to connect Lenses to an Amazon MSK Serverless cluster.

It is recommended to install the Agent on an EC2 instance or with EKS in the same VPC as your MSK Serverless cluster.

Security Groups

Enable communications between the Agent & the Amazon MSK Serverless cluster by opening the Amazon MSK Serverless cluster's security group in the AWS Console and add the IP address of your Agent installation.

IAM Policy

To authenticate the Agent & access resources within our MSK Serverless cluster, we'll need to create an IAM policy and apply that to the resource (EC2, EKS cluster, etc) running the Agent service. here is an example IAM policy with sufficient permissions which you can associate with the relevant IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:topic/[cluster_name]/[cluster_uuid]/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:[region]:[aws_account_id]:group/[cluster_name]/[cluster_uuid]/*"
        }
    ]
}

MSK Serverless IAM to be used after cluster creation. Update this IAM policy with the relevant ARN.

Select your MSK endpoint

Click your MSK Serverless Cluster in the MSK console and select View Client Information page to check the bootstrap server endpoint.

Configure Provisioning

provisioning.yaml

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandle

SQL Processor Configurations

To enable the creation of SQL Processors that create consumer groups, you need to add the following statement in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Topic*",
    "kafka-cluster:WriteData",
    "kafka-cluster:ReadData"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to add the following statement for the registries and schemas in your IAM policy:

{
  "Action": [
    "kafka-cluster:*Group*"
  ],
  "Resource": "arn:aws:kafka:[region]:[aws_account_id]:cluster/[cluster_name]/[cluster_uuid]/*"
}

Update the placeholders in the IAM policy based on the relevant MSK Serverless cluster ARN.

To integrate with the AWS Glue Schema Registry, you also need to modify the security policy for the registry and schemas, which results in additional functions within it:

{
  "Action": [
    "glue:DeregisterDataPreview",
    "glue:ListRegistries",
    "glue:CreateRegistry",
    "glue:RegisterSchemaVersion",
    "glue:GetRegistry",
    "glue:UpdateRegistry",
    "glue:ListSchemas",
    "glue:DeleteRegistry",
    "glue:GetSchema",
    "glue:CreateSchema",
    "glue:ListSchemaVersions",
    "glue:GetSchemaVersion",
    "glue:UpdateSchema",
    "glue:DeleteSchemaVersions"
  ],
  "Resource": [
    "arn:aws:glue:[region]:[aws_account_id]:registry/*",
    "arn:aws:glue:[region]:[aws_account_id]:schema/*"
  ]
}

More details about how IAM works with MSK Serverless can be found in the documentation: MSK Serverless

Limitations

When using the Agent with MSK Serverless:

The agent does not receive Prometheus-compatible metrics from the brokers because they are not exported outside of CloudWatch.
The agent does not configure quotas and ACLs because MSK Serveless does not allow this.

Azure EventHubs

This page describes connection Lenses to Azure EventHubs.

Create a Data Integration API key

Add a shared access policy
- Navigate to your Event Hub resource and select Shared access policies in the Settings section.
- Select + Add shared access policy, give a name, and check all boxes for the permissions (Manage, Send, Listen)
Once the policy is created, obtain the Primary Connection String, by clicking the policy and copying the connection string. The connection string will be used as a JAAS password to connect to Kafka.
The bootstrap broker [YOUR_EVENT_HUBS_NAMESPACE].servicebus.windows.net:9093

Configure Provisioning

Set the following in the provisioning.yaml

First set environment variable

Note that "\" at "$ConnectionString" is set additionally to escape the $ sign.

terminal

export SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="\$ConnectionString" password="Endpoint=sb://[SB_URL]/;SharedAccessKeyName=[KEY_NAME];SharedAccessKey=[ACCESS_KEY]";

provision.yaml

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
    saslJaasConfig:
      value: '${SASL_JAAS_CONFIG}'
    saslMechanism:
      value: PLAIN
    protocol:
      value: SASL_SSL

provision.yaml

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: [my-tag]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
          - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
      saslJaasConfig:
        value: org.apache.kafka.common.security.plain.PlainLoginModule required username="\$ConnectionString" password="Endpoint=sb://[SB_URL]/;SharedAccessKeyName=[KEY_NAME];SharedAccessKey=[ACCESS_KEY]";
      saslMechanism:
        value: PLAIN
      protocol:
        value: SASL_SSL

Azure HDInsight

This page describes connection Lenses to a Azure HDInsight cluster.

Find your Kafka endpoints

In our Azure Portal, go to Dashboards->Ambari home. Go to Kafka->Configs->Kafka Broker->Kafka Broker hosts

Optionally find your Zookeeper endpoints

Optionally get the Zookeeper endpoints: Go to Zookeeper->Configs->Zookeeper Server->Zookeeper Server hosts.

Configure Provisioning

See for different configurations for your security protocols.

kafka:
- name: Kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

Confluent Cloud

This page describes configuring Lenses to connect to Confluent Cloud.

For Confluent Platform see Apache Kafka.

Create a Data Integration API key

From Data integration API keys, select Create Key.
For this guide select Global access to get your API Key and API Secret Key.
Go to Cluster Settings to get your Bootstrap Server.

Configure Provisioning

Set the following in the provisioning.yaml

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required 
        username="[YOUR_API_KEY]" 
        password="[YOUR_API_KEY_SECRET]";

Confluent Platform

This page describes configuring Lenses to connect to Confluent Platform.

For Confluent Platform see Apache Kafka.

IBM Event Streams

This page describes how to connect Lenses to IBM Event Streams.

IBM Event Streams requires a replication factor of 3. Ensure you set the replication factor accordingly for Lenses internal topics.

See .

Find your bootstrap endpoints

From the IBM Cloud console, locate your bootstrap_endpoints, for the service credentials you want to connect with.

Configure Provisioning

Set the following in the provisioning.yaml:

Use "token" as the username in the Jaas Config. Set the password as your API KEY from IBM Event streams

provisioning.yaml

kafka:
- name: Kafka
  version: 1
  tags: ['my-tag']
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_ENDPOINTS]
    protocol: 
      value: SASL_SSL
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required 
        username="token" 
        password="[YOUR_API_KEY]";

Schema Registries

This page describes adding a Schema Registries to the Lenses Agent.

Overview

This page describes an overview of connecting a Lenses Agent with Schema Registries

Consider if you have a high number of schemas.

Authentication

TLS and basic authentication are supported for connections to Schema Registries.

JMX Metrics

The Agent can collect Schema registry metrics via:

JMX
Jolokia

Supported formats

AVRO
PROTOBUF

JSON and XML formats are supported by Lenses but without a backing schema registry.

Schema deletion

To enable the deletion of schemas in the UI, set the following in the lenses.conf file.

lenses.conf

## Enable schema deletion in the Lenses UI
## default: false
lenses.schema.registry.delete = true

## When a topic is deleted,
## automatically delete also its associated Schema Registry subjects
## default: false
lenses.schema.registry.cascade.delete = true

IBM Event Streams supports hard deletes only

AWS Glue

This page describes connection to AWS Glue.

AWS Glue Schema Registry connection, depends on an .

Set the following examples in provisioning.yaml

These are examples of provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it.

Using AWS Access Key

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Access Key
      accessKeyId: 
        value: my-access-key-id
      secretAccessKey: 
        value: my-secret-access-key
      region: 
        value: eu-west-1
      
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        reference: my-aws-connection
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
          value: arn:aws:glue:region:123123123:registry/my-registry

Using AWS Credentials Chain

aws:
  - name: my-aws-connection
    version: 1
    tags: []
    configuration:
      region:
        value: eu-north-1
      authMode:
        value: "Credentials Chain"

glueSchemaRegistry:
  - name: schema-registry
    version: 1
    tags: []
    templateName: SchemaRegistry
    configuration:
      authMode:
        reference: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:eu-north-1:[account-id]:registry/[name]

Confluent

This page describes connecting Lenses to Confluent schema registries.

Set the following examples in provisioning.yaml

Simple configuration, with JMX metrics

The URLs (nodes) should always have a scheme defined (http:// or https://).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      ## all metrics properties are optional
      metricsPort: 
        value: 9581
      metricsType: 
        value: JMX
      metricsSsl: 
        value: false

Basic authentication

For Basic Authentication, define username and password properties.

confluentSchemaRegistry:
- name: schema-registry
  tags: ["tag1"]
  version: 1    
  configuration:
    schemaRegistryUrls:
      value:
        - http://my-sr.host1:8081
        - http://my-sr.host2:8081
    username: 
      value: my-username
    password: 
      value: my-password

TLS with custom truststore

A custom truststore is needed when the Schema Registry is served over TLS (encryption-in-transit) and the Registry’s certificate is not signed by a trusted CA.

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslKeystore:
        fileRef:
          filePath: /path/to/my/keystore.jks
      sslKeystorePassword: 
        value: myPassword

Hard or soft delete

By default, Lenses will use hard delete for Schema Registry. To use soft delete, add the following property:

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      hardDelete:
        value: true

Apicurio

This page describes connecting Lenses to Apicurio.

Apicuro supports the following versions of Confluent's API:

Confluent Schema Registry API v6
Confluent Schema Registry API v7

Set the following examples in provisioning.yaml

Set the schema registry URLs to include the compatibility endpoints, for example:

http://localhost:8080/apis/ccompat/v6

provisioning.yaml

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://localhost:8080/apis/ccompat/v6

IBM Event Streams Registry

This page describes connecting Lenses to IBM Event Streams schema registry.

Requires Enterprise subscription on IBM Event Streams and only hard delete is supported for IBM Event streams

To configure an application to use this compatibility API, specify the Schema Registry endpoint in the following format:

https://token:{$APIKEY}@{$HOST}/confluent

Use "token" as the username. Set the password as your API KEY from IBM Event streams

Set the following examples in provisioning.yaml

provisioning.yaml

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - https://token:{$APIKEY}@{$HOST}/confluent

Kafka Connect

This page describes adding a Kafka Connect Cluster to the Lenses Agent.

Lenses integrates with Kafka Connect Clusters to manage connectors.

The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Multiple Kafka Connect clusters are supported.

If you are using Kafka Connect < 2.6 set the following to ensure you can see Connectors

lenses.features.connectors.topics.via.api.enabled=false

Consider Rate Limiting if you have a high number of connectors.

Simple configuration, with JMX metrics

The URLs (workers) should always have a scheme defined (http:// or https://).

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    version: 1    
    tags: ["tag1"]
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083
      metricsPort: 
        value: 9585
      metricsType: 
        value: JMX

Basic authentication

For Basic Authentication, define username and password properties.

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      username: 
        value: my-username
      password: 
        value: my-password

TLS with custom truststore

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      sslTruststore:
        file: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

provisioning.yaml

connect:
  name: my-connect-cluster-name
  tags: ["tag1"]
  version: 1    
  configuration:
    workers:
      value:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083    
    sslKeystore:
      file: /path/to/my/keystore.jks
    sslKeystorePassword: 
      value: myPassword

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

lenses.conf

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]

Zookeeper

This page describes adding a Zookeeper to the Lenses Agent.

Set the following examples in provisioning.yaml

Simple configuration, without metrics

zookeeper:
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds

Simple configuration, with JMX metrics

Simple configuration with Zookeeper metrics read via JMX.

zookeeper:    
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false

With such a configuration, Lenses will use 3 Zookeeper nodes and will try to read their metrics from following URLs (notice the same port - 9581 - used for all of them, as defined by metricsPort property):

my-zookeeper-host-0:9581
my-zookeeper-host-1:9581
my-zookeeper-host-2:9581

AWS

Add a connection to AWS in the Lenses Agent.

The agent uses an AWS in three places:

AWS IAM connection to MSK for Lenses itself
Connecting to AWS Glue
Alert channels to Cloud Watch.

If the Agent is deployed on an EC2 Instance or has access to AWS credentials in the default AWS toolchain that can be used instead.

provisioning.yaml

aws:
- name: my-aws-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # Way to authenticate against AWS.Credentials Chain or Access Key
    authMode:
      value:
    # Access key ID of an AWS IAM account.
    accessKeyId:
       value:
    # Secret access key of an AWS IAM account.
    secretAccessKey:
       value:
    # AWS region to connect to. If not provided, this is deferred to client 
    # configuration.
    region:
       value:
    # Specifies the session token value that is required if you are using temporary 
    # security credentials that you retrieved directly from AWS STS operations.
    sessionToken:
       value:

Alert & Audit integrations

Connect the Lenses Agent to your alerting and auditing systems.

The Agent can send out alerts and audits events. Once you have configured alert and audit connections, you can create alert and audit channels to route events to them.

Alerts

DataDog

provisioning.yaml

datadog:
- name: my-datadog-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The Datadog site.
    site:
      value:
    # The Datadog API key.
    apiKey:
      value:   
    # The Datadog application key.
    applicationKey:
      value:

AWS CloudWatch

See AWS connection.

PagerDuty

provisioning.yaml

pagerduty:
- name: my-pagerduty-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # An Integration Key for PagerDuty's service with Events API v2 integration type.
    integrationKey:
      value:

Slack

provisioning.yaml

slack:
- name: my-slack-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The Slack endpoint to send the alert to.
    webhookUrl:
      value:

Alert Manager

provisioning.yaml

alertManager:
- name: my-alertmanager-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # Comma separated list of Alert Manager endpoints.
    endpoints:
      value:

Webook (Email, SMS, HTTP and MS Teams)

provisioning.yaml

webhook:
- name: my-webhook-alert-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The host name for the HTTP Event Collector API of the Splunk instance.
    host:
      value: 
    # The port number for the HTTP Event Collector API of the Splunk instance. (int)
    port:
      value:  
    # Set to true in order to set the URL scheme to https. 
    # Will otherwise default to http.
    useHttps:
      value:
    # An array of (secret) strings to be passed over to alert channel plugins.
    creds:
      value:
        - 
        -

Audits

Webhook

provisioning.yaml

webhook:
- name: my-webhook-audit-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The host name for the HTTP Event Collector API of the Splunk instance.
    host:
      value: 
    # The port number for the HTTP Event Collector API of the Splunk instance. (int)
    port:
      value:  
    # Set to true in order to set the URL scheme to https. 
    # Will otherwise default to http.
    useHttps:
      value:
    # An array of (secret) strings to be passed over to alert channel plugins.
    creds:
      value:
        - 
        -

Splunk

provisioning.yaml

splunk:
- name: my-splunk-connection
  version: 1
  tags: [tag1, tag2]
  configuration:
    # The host name for the HTTP Event Collector API of the Splunk instance.
    host:
      value: 
    # The port number for the HTTP Event Collector API of the Splunk instance. (int)
    port:
      value:  
    # Use TLS. Boolean, default false
    useHttps:
      value:
    # This is not encouraged but is required for a Splunk Cloud Trial instance. Bool
    insecure:
      value:
    # HTTP event collector authorization token. (string)
    token:
      value:

Hardware & OS

This page describes the hardware and OS prerequisites for Lenses.

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Memory & CPU

This page describes the memory & cpu prerequisites for Lenses.

This documentation provides memory recommendations for Lenses.io, considering the number of Kafka topics, the number of schemas, and the complexity of these schemas (measured by the number of fields). Proper memory allocation ensures optimal performance and stability of Lenses.io in various environments.

Key Considerations

Number of Topics: Kafka topics require memory for indexing, metadata, and state management.
Schemas and Their Complexity: The memory impact of schemas is influenced by both the number of schemas and the number of fields within each schema. Each schema field contributes to the creation of Lucene indexes, which affects memory usage.

Baseline Memory Requirements

For a basic setup with minimal topics and schemas:

Minimum Memory: 4 GB
Recommended Memory: 8 GB

This setup assumes:

Fewer than 100 topics
Fewer than 100 schemas
Small schemas with few fields (less than 10 fields per schema)

Scaling with Topics

Memory requirements increase with the number of topics. Topics are used as the primary reference for memory scaling, with additional considerations for schemas.

Impact of Schemas and Their Complexity

Schemas have a significant impact on memory usage, particularly as the number of fields within each schema increases. The memory impact is determined by both the number of schemas and the complexity (number of fields) of these schemas.

Memory Addition Based on Schema Complexity

Cross-Reference Table for Topics and Schema Complexity

Example Configurations

To help illustrate how to apply these recommendations, here are some example configurations considering both topics and schema complexity:

Small Setup

Topics: 500
Schemas: 100 (average size 50 KB, 8 fields per schema)

Recommended Memory: 8 GB

Schema Complexity: Low → No additional memory needed.

Total Recommended Memory: 8 GB

Medium Setup

Topics: 5,000
Schemas: 1,000 (average size 200 KB, 25 fields per schema)

Base Memory: 12 GB

Schema Complexity: Moderate → No additional memory needed.

Total Recommended Memory: 16 GB

Large Setup

Topics: 15,000
Schemas: 3,000 (average size 500 KB, 70 fields per schema)

Base Memory: 32 GB

Schema Complexity: High → Add 3 GB for schema complexity.

Total Recommended Memory: 35 GB

High-Volume Setup Examples

30,000 Topics
- Schemas: 5,000 (average size 300 KB, 30 fields per schema)
Base Memory: 64 GB
- Schema Complexity: Moderate → Add 5 GB for schema complexity.
Total Recommended Memory: 69 GB

Additional Considerations

High Throughput: If your Kafka cluster is expected to handle high throughput, consider adding 20-30% more memory than the recommendations.
Complex Queries and Joins: If using Lenses.io for complex data queries and joins, consider increasing the memory allocation by 10-15% to accommodate the additional processing.
Monitoring and Adjustment: Regularly monitor memory usage and adjust based on actual load and performance.

Conclusion

Proper memory allocation is crucial for the performance and reliability of Lenses.io, especially in environments with a large number of topics and complex schemas. While topics provide a solid baseline for memory recommendations, the complexity of schemas—particularly the number of fields—can also significantly impact memory usage. Regular monitoring and adjustments are recommended to ensure that your Lenses.io setup remains performant as your Kafka environment scales.

Database

This page describes configuring the database connection for the Lenses Agent.

Once you have created a role for the agent to use you can then configure the Agent in the lenses.conf file:

lenses.conf

lenses.storage.postgres.host="my-postgres-server"
lenses.storage.postgres.port=5432
lenses.storage.postgres.username="lenses_agent"
lenses.storage.postgres.database="lenses_agent"
lenses.storage.postgres.password="changeme"

Additional configurations for the PostgreSQL database connection can be passed under the lenses.storage.postgres.properties configuration prefix.

One Postgres server can be used for all agents by using a separate database or schema each.

For the Agent see lenses.storage.postgres.schema or lenses.storage.postgres.database

The supported parameters can be found in the PostgreSQL documentation. For example:

lenses.conf

# require SSL encryption with full host verification
lenses.storage.postgres.properties.ssl=true
lenses.storage.postgres.properties.sslmode="verify-full"
lenses.storage.postgres.properties.sslcert="/path/to/certs/lenses.crt.pem"
lenses.storage.postgres.properties.sslkey="/path/to/certs/lenses.key.pk8"
lenses.storage.postgres.properties.sslpassword="mypassword"
lenses.storage.postgres.properties.sslrootcert="/path/to/certs/CA.crt.pem"

Database Role

terminal

# login as superuser and add Lenses role and database
psql -U postgres -d postgres <<EOF
CREATE ROLE lenses_agent WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_agent OWNER lenses_agent;
EOF

Connection pooling

The Agent uses the HikariCP library for high-performance database connection pooling.

The default settings should perform well but can be overridden via the lenses.storage.hikaricp configuration prefix. The supported parameters can be found in the HikariCP documentation.

Camelcase configuration keys are not supported in agent configuration and should be translated to dot notation.

For example:

# set maximumPoolSize to 25
lenses.storage.hikaricp.maximum.pool.size=25

TLS

This page describes how to configure TLS for the Lenses Agent.

By default, the Agent does not provide TLS termination but can be enabled via a configuration option. TLS termination is recommended for enhanced security and a prerequisite for integrating with SSO (Single Sign On) via SAML2.0.

TLS termination can be configured directly within Agent or by using a TLS proxy or load balancer.

Global Truststore

To use a non-default global truststore, set the path in accordingly with the LENSES_OPTS variable.

LENSES_OPTS=-Djavax.net.ssl.trustStore=/path/to/truststore

Custom Truststore

lenses.conf

lenses.ssl.truststore.location = "/path/to/truststore.jks"
lenses.ssl.truststore.password = "changeit"

Mutual TLS

To enable mutual TLS, set your keystore accordingly.

lenses.conf

# To secure and encrypt all HTTPS connections to Lenses via TLS termination.
# Java Keystore location and passwords
lenses.ssl.client.auth = true
lenses.ssl.keystore.location = "/path/to/keystore.jks"
lenses.ssl.keystore.password = "changeit"
lenses.ssl.key.password      = "changeit"


# You can also tweak the TLS version, algorithm and ciphers
#lenses.ssl.enabled.protocols = "TLSv1.2"
#lenses.ssl.cipher.suites     = "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WIT

Kafka ACLs

This page describes the Kafka ACLs prerequisites for the Lenses Agent if ACLs are enabled on your Kafka clusters.

These ACLs are for the underlying Lenses Agent Kafka client. Lenses has its own set of permissions guarding access.

When your Kafka cluster is configured with an authorizer which enforces ACLs, the Agent will need a set of permissions to function correctly.

Common practice is to give teh Agent superuser status or the complete list of available operations for all resources. The IAM model of Lenses can then be used to restrict the access level per user.

kafka-acls \
    --bootstrap-server [broker.url:9092] --command-config [client.properties] \
    --add \
    --allow-principal [User:Lenses] \
    --allow-host [lenses.host] \
    --operation All \
    --topic * \
    --group * \
    --delegation-token * \
    --cluster

Minimal Permissions

The Agent needs permission to manage and access their own internal Kafka topics:

__topology
__topology__metrics

kafka-acls \
    --bootstrap-server [broker.url:9092] --command-config [client.properties] \
    --add \
    --allow-principal [User:Lenses] \
    --allow-host [lenses.host] \
    --operation All \
    --topic [topic]

It also needs to read and describe permissions for the consumer offsets and Kafka Connect topics —if enabled:

__consumer_offsets
connect-configs
connect-offsets
connect-status

kafka-acls \
    --bootstrap-server [broker.url:9092] --command-config [client.properties] \
    --add \
    --allow-principal [User:Lenses] \
    --allow-host [lenses.host] \
    --operation Describe \
    --operation DescribeConfigs \
    --operation Read \
    --topic [topic]

This same set of permissions is required for any topic that the agent must have read access.

kafka-acls \
    --bootstrap-server [broker.url:9092] --command-config [client.properties] \
    --add \
    --allow-principal [User:Lenses] \
    --allow-host [lenses.host] \
    --operation Describe \
    --operation DescribeConfigs \
    --operation Read \
    --topic *

DescribeConfigs was added in Kafka 2.0. It may not be needed for versions before 2.2.

Additional permissions are needed to produce topics or manage them.

Consumer Groups

Permission to at least read and describe consumer groups is required to take advantage of the Consumer Groups' monitoring capabilities.

kafka-acls \
    --bootstrap-server [broker.url:9092] --command-config [client.properties] \
    --add \
    --allow-principal [User:Lenses] \
    --allow-host [lenses.host] \
    --operation Describe \
    --operation Read \
    --group *

Additional permissions are needed to manage groups.

ACLs

To manage ACLs, permission to the cluster is required:

kafka-acls \
    --bootstrap-server [broker.url:9092] --command-config [client.properties] \
    --add \
    --allow-principal [User:Lenses] \
    --allow-host [lenses.host] \
    --operation Describe \
    --operation DescribeConfigs \
    --operation Alter \
    --cluster

Rate Limiting

Rate limit the calls the Lenses Agent makes to Schema Registries and Connect Clusters.

To rate limit the calls the Agent makes to Schema Registries or Connect Clusters set the following the Agent configuration:

# schema registry
lenses.schema.registry.client.http.rate.type="sliding" 
lenses.schema.registry.client.http.rate.maxRequests= 200
lenses.schema.registry.client.http.rate.window="2 seconds"

# connect clusters
lenses.connect.client.http.rate.type="sliding"                 
lenses.connect.client.http.rate.maxRequests=200        
lenses.connect.client.http.rate.window="2 seconds"

The exact values provided will depend on your setup, for example the number of schemas, how often are new schemas added, so some trial and error is required.

JMX Metrics

This page describes the how to retrieve Lenses Agent JMX metrics.

The JMX endpoint is managed by the lenses.jmx.port option. To disable the JMX leave the option empty.

To enable monitoring of the Agent metrics:

LENSES_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=[HOSTNAME]"

To export via Prometheus exporter:

export LENSES_OPTS="-javaagent:/path/to/jmx_exporter/fastdata_agent.jar=9102:/path/to/jmx_exporter/client.yml"

The Agent Docker image (lensesio/lenses) automatically sets up the Prometheus endpoint. You only have to expose the 9102 port to access it.

Setting up the JMX Agent with Basic Auth.

This will be done in two parts. The first part is about setting up the required files that JMX Agent will require and the second is about the options we need to pass to the agent.

Setting up required files

First let’s create a new folder called jmxremote

mkdir -vp /etc/jmxremote

To enable basic auth JMX, first create two files:

jmxremote.access
jmxremote.password

JMX Password file

The password file has the credentials that the JMX agent will check during client authentication

cat /etc/jmxremote/jmxremote.password 
admin admin
guest admin

The above code is registering 2 users.

UserA:
- username admin
- password admin
UserB:
- username: guest
- password: admin

JMX Access file

The access file has authorization information, like who is allowed to do what.

cat jmxremote/jmxremote.access 
admin readwrite
guest readonly

In the above code, we can see that the admin user can do read and write operations in JMX, while guest user can only read the JMX content.

Enable JMX with Basic Auth Protection

Now, to enable JMX with basic auth protection, all we need to do is pass the following options in the JRE’s env that will run the Java process you need to protect the jmx.

Let’s assume this java process is Kafka.

Change the permissions on both files so only owner can edit and view them.

chmod -R 0600 /etc/jmxremote
chown -R <user-that-will-run-kafka-name>:<user-that-will-run-kafka-group> /etc/jmxremote/jmxremote.*

If you do not change the permissions to 0600 and to the user that will run the jre process, then JMX will Agent will cause an error complaining that the Process is not the owner of the files that will be used for authentication and authorization.

Finally export the following options in the user’s env which will run Kafka.

export BROKER_JMX_OPTS= "-Dcom.sun.management.jmxremote=true \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.local.only=false \
  -Djava.rmi.server.hostname=10.15.3.1 \
  -Dcom.sun.management.jmxremote.rmi.port=9581 \
  -Dcom.sun.management.jmxremote.access.file=/etc/jmxremote/jmxremote.access \
  -Dcom.sun.management.jmxremote.password.file=/etc/jmxremote/jmxremote.password \
  -Dcom.sun.management.jmxremote.port=9581

Secure JMX with TLS Encryption

First setup JMX with basic auth as shown in the Secure JMX: Basic Auth page.

To enable TLS Encryption/Authentication in JMX you need a jks keystore and truststore.

Please note that both JKS Truststore and Keystore should have the same password.

The reason for this is because the javax.net.ssl class will use the password you pass to the Keystore as the keypassword

Let’s assume this java process is Kafka and that you have installed the keystore.jks and truststore.jks under `/etc/certs``

Export the following options in the user’s env which will run Kafka.

export BROKER_JMX_OPTS= "-Dcom.sun.management.jmxremote=true
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=true \
  -Dcom.sun.management.jmxremote.local.only=false \
  -Djava.rmi.server.hostname=10.15.3.1 \
  -Dcom.sun.management.jmxremote.rmi.port=9581 \
  -Dcom.sun.management.jmxremote.access.file=/etc/jmxremote.access \
  -Dcom.sun.management.jmxremote.password.file=/etc/jmxremote.password \
  -Dcom.sun.management.jmxremote.port=9581 \
  -Djavax.net.ssl.keyStore=/etc/certs/kafka.jks \
  -Djavax.net.ssl.keyStorePassword=somePassword \
  -Djavax.net.ssl.trustStore=/etc/certs/truststore.jks \
  -Djavax.net.ssl.trustStorePassword=somePassword \
  -Dcom.sun.management.jmxremote.registry.ssl=true \
  -Dcom.sun.management.jmxremote.ssl.need.client.auth=true

SQL Processor Deployment

This page describes how to configure the agent to deploy and manage SQL Processors for stream processing.

Lenses can be used to define & deploy stream processing applications that read from Kafka and write back to Kafka with SQL. They are based on the Kafka Stream framework. They are known as SQL Processors.

SQL processing of real-time data can run in 2 modes:

SQL In-Process - the workload runs inside of the Lenses Agent.
SQL in Kubernetes - the workload runs & scale on your Kubernetes cluster.

Which mode the SQL Processors will run as should be defined within the lenses.conf before Lenses is started.

In-Process Mode

In this mode, SQL processors run as part of the Agent process, sharing resources, memory, and CPU time with the rest of the platform.

This mode of operation is meant to be used for development only.

As such, the agent will not allow the creation of more than 50 SQL Processors in In Process mode, as this could impact the platform's stability and performance negatively.

For production, use the KUBERNETES mode for maximum flexibility and scalability.

Set the execution configuration to IN_PROC

# Set up Lenses SQL processing engine
lenses.sql.execution.mode = "IN_PROC"

Set the directory to store the internal state of the SQL Processors:

lenses.sql.state.dir = "/tmp/sql-kstream-state"

TLS connections to Kafka and Schema Registries

SQL processors use the same connection details that Agent uses to speak to Kafka and Schema Registry. The following properties are mounted, if present, on the file system for each processor:

Kafka
1. SSLTruststore
2. SSLKeystore
Schema Registry
1. SSL Keystore
2. SSL Truststore

The file structure created by applications is the following: /run/[lenses_installation_id]/applications/

Keep in mind Lenses require an installation folder with write permissions. The following are tried:

/run
/tmp

Kubernetes Mode

Kubernetes can be used to deploy SQL Processors. To configure Kubernetes, set the mode to KUBERNETES and configure the location of the kubeconfig file.

When the Agent is deployed inside Kubernetes, the lenses.kubernetes.config.file configuration entry should be set to an empty string. The Kubernetes client will auto-configure from the pod it is deployed in.

The SQL Processor docker image is live in Dockerhub.

lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = ""  # Only needed if you use a custom image

# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000

# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000

Custom Serdes

Custom serdes should be embedded in a new Lenses SQL processor Docker image.

To build a custom Docker image, create the following directory structure:

mkdir -p processor-docker/serde

Copy your serde jar files under processor-docker/serde.

Create Dockerfile containing:

FROM lensesioextra/sql-processor:4.2

ADD serde /opt/serde
ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde

Build the Docker.

cd processor-docker
docker build -t example/lsql-processor .

Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):

lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"

Don't use the LPFP_ prefix.

Internally, Lenses prefixes all its properties with LPFP_.

Avoid passing custom environment variables starting with LPFP_ as it may cause the processors to fail.

Use Role/RoleBinging to deploy Lenses processors

To deploy Lenses Processors in Kubernetes the suggested way is to activate RBAC in Cluster level through Helm values.yaml:

rbacEnable: true

If you want to limit the permissions Lenses has against your Kubernetes cluster, you can use Role/RoleBinding resources instead.

To achieve this you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: [ROLE_NAME]
  namespace: [PROCESSORS_NAMESPACE]
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: [ROLE_BINDING_NAME]
  namespace: [PROCESSOR_NAMESPACE]
subjects:
- kind: ServiceAccount
  namespace: [LENSES_NAMESPACE]
  name: [SERVICE_ACCOUNT_NAME]
roleRef:
  kind: Role
  name: [ROLE_NAME]
  apiGroup: rbac.authorization.k8s.io

example for:

Lenses namespace = lenses-ns
Processor namespace = lenses-proc-ns

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role
  namespace: lenses-proc-ns
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role-binding
  namespace: lenses-proc-ns
subjects:
- kind: ServiceAccount
  namespace: lenses-ns
  name: default
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io

You can repeat this for as many namespaces you may want Lenses to have access to.

Finally you need to define in Lenses configuration which namespaces can Lenses access. To achieve this amend values.yaml to contain the following:

lenses:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "[PROCESSORS NAMESPACE]"
        ]
      }

example:

lenses:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "lenses-processors"
        ]
      }

Logs

This page describes configuring Lenses Agent logging.

Changes to the logback.xml are hot reloaded by the Agent, no need to restart.

All logs are emitted unbuffered as a stream of events to both stdout and to rotating files inside the directory logs/.

The logback.xml file is used to configure logging.

If customization is required, it is recommended to adapt the default configuration rather than write your own from scratch.

The file can be placed in any of the following directories:

the directory where the Agent is started from
/etc/lenses/
agent installation directory.

The first one found, in the above order, is used, but to override this and use a custom location, set the following environment variable:

export LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/path/to/logback.xml"

Default configuration

The default configuration file is set up to hot-reload any changes every 30 seconds.

Log Level

The default log level is set to INFO (apart from some very verbose classes).

Log Format

All the log entries are written to the output using the following pattern:

%d{ISO8601} %-5p [%c{2}:%L] [%thread] %m%n

You can adjust this inside logback.xml to match your organization’s defaults.

Log Location

logs/ you will find three files: lenses.log, lenses-warn.log and metrics.log. The first contains all logs and is the same as the stdout. The second contains only messages at level WARN and above. The third one contains timing metrics and can be useful for debugging.

Log Buffering

The default configuration contains two cyclic buffer appenders: "CYCLIC-INFO” and “CYCLIC-METRICS”. These appenders are required to expose the Agent logs within the Admin UI.

Plugins

This page describes how to install plugins in the Lenses Agent.

The following implementations can be specified:

Serializers/Deserializers Plug your serializer and deserializer to enable observability over any data format (i.e., protobuf / thrift)
Custom authentication Authenticate users on your proxy and inject permissions HTTP headers.
LDAP lookup Use multiple LDAP servers or your group mapping logic.
SQL UDFs User Defined Functions (UDF) that extend SQL and streaming SQL capabilities.

Once built, the jar files and any plugin dependencies should be added to the Agent and, in the case of Serializers and UDFs, to the SQL Processors if required.

Adding plugins

On startup, the Agent loads plugins from the $LENSES_HOME/plugins/ directory and any location set in the environment variable LENSES_PLUGINS_CLASSPATH_OPTS. The Agent is watching, and dropping a new plugin will hot-reload it. For the Agent docker (and Helm chart) you use /data/plugins.

Any first-level directories under the paths mentioned above, detected on startup will also be monitored for new files. During startup, the list of monitored locations will be shown in the logs to help confirm the setup.

...
Initializing (pre-run) Lenses
Installation directory autodetected: /opt/lenses
Current directory: /data
Logback configuration file autodetected: logback.xml
These directories will be monitored for new jar files:
 - /opt/lenses/plugins
 - /data/plugins
 - /opt/lenses/serde
Starting application
...

Whilst all jar files may be added to the same directory (e.g /data/plugins), it is suggested to use a directory hierarchy to make management and maintenance easier.

An example hierarchy for a set of plugins:

├── security
│   └── sso_header_decoder.jar
├── serde
│   ├── protobuf_actions.jar
│   └── protobuf_clients.jar
└── udf
    ├── eu_vat.jar
    ├── reverse_geocode.jar
    └── summer_sale_discount.jar

SQL Processors in Kubernetes

There are two ways to add custom plugins (UDFs and Serializers) to the SQL Processors; (1) via making available a tar.gz archive at an HTTP (s) address, or (2) via creating a custom docker image.

Archive served via HTTP

With this method, a tar archive, compressed with gzip, can be created that contains all plugin jars and their dependencies. Then this archive should be uploaded to a web server that the SQL Processors containers can access, and its address should be set with the option lenses.kubernetes.processor.extra.jars.url.

Step by step:

Create a tar.gz file that includes all required jars at its root:
```
tar -czf [FILENAME.tar.gz] -C /path/to/jars/ *
```
Upload to a web server, ie. https://example.net/myfiles/FILENAME.tar.gz

Set

lenses.kubernetes.processor.extra.jars.url=https://example.net/myfiles/FILENAME.tar.gz

For the docker image, set the corresponding environment variable

LENSES_KUBERNETES_PROCESSOR_EXTRA_JARS_URL=https://example.net/myfiles/FILENAME.tar.gz`

Custom Docker image

The SQL Processors inside Kubernetes use the docker image lensesio-extra/sql-processor. It is possible to build a custom image and add all the required jar files under the /plugins directory, then set lenses.kubernetes.processor.image.name and lenses.kubernetes.processor.image.tag options to point to the custom image.

Step by step:

Create a Docker image using lensesio-extra/sql-processor:VERSION as a base and add all required jar files under /plugins:
```
FROM lensesio-extra/sql-processor:4.2
ADD jars/* /plugins
```
```
docker build -t example/sql-processor:4.2 .
```
Upload the docker image to a registry:
```
docker push example/sql-processor:4.2
```

Set

lenses.kubernetes.processor.image.name=example/sql-processor
lenses.kubernetes.processor.image.tag=4.2

For the docker image, set the corresponding environment variables

LENSES_KUBERNETES_PROCESSOR_IMAGE_NAME=example/sql-processor
LENSES_KUBERNETES_PROCESSOR_IMAGE_TAG=4.2