Lenses can be deployed in the following ways:

To install Lenses from the archive you must:

1. Extract the archive

2. Configure Lenses

3. Start Lenses

Extracting the archive

Extract the archive using the following command

Inside the extract archive, you will find.

Starting Lenses

Start Lenses by running:

or pass the location of the config file:

If you do not pass the location of the config file, Lenses will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the security.conf to be readable only by the lenses user.

The agent needs write access in 4-5 places in total:

1. [RUNTIME DIRECTORY] When Lenses runs, it will create at least one directory under the directory it is run in:

   1. [RUNTIME DIRECTORY]/logs Where logs are stored

   2. [RUNTIME DIRECTORY]/logs/lenses-sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.

   3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.

   4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.

   5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

JNI libraries

Lenses and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

1. Mount /tmp without noexec

2. or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

SystemD example

If your server uses systemd as a Service Manager, then manage Lenses (start upon system boot, stop, restart). Below is a simple unit file that starts Lenses automatically on system boot.

Global Truststore

Lenses uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, Secure LDAP, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (e.g. Secure LDAP and JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

Hardware & OS

Run on any Linux server. For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Increase as a super-user the soft limit to 4096 with:

Use 6GB RAM/4 CPUs and 500MB disk space.

Simple configuration, without metrics

zookeeper:
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds

Simple configuration, with JMX metrics

Simple configuration with Zookeeper metrics read via JMX.

zookeeper:    
- name: Zookeeper
  version: 1
  tags: ["tag1"]
  configuration:
    zookeeperUrls:
      value:
        - my-zookeeper-host-0:2181
        - my-zookeeper-host-1:3181
        - my-zookeeper-host-2:4181
    # optional, a suffix to Zookeeper's connection string
    zookeeperChrootPath: 
      value: "/mypath" 
    zookeeperSessionTimeout: 
      value: 10000 # in milliseconds
    zookeeperConnectionTimeout: 
      value: 10000 # in milliseconds
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false

With such a configuration, Lenses will use 3 Zookeeper nodes and will try to read their metrics from following URLs (notice the same port - 9581 - used for all of them, as defined by metricsPort property):

• my-zookeeper-host-0:9581

• my-zookeeper-host-1:9581

• my-zookeeper-host-2:9581

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582

Custom url mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

• my-kafka-host-0:9582 - because of metricsCustomUrlMappings

• my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

Confluent

Simple configuration, with JMX metrics

The URLs (nodes) should always have a scheme defined (http:// or https://).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      ## all metrics properties are optional
      metricsPort: 
        value: 9581
      metricsType: 
        value: JMX
      metricsSsl: 
        value: false

Basic authentication

For Basic Authentication, define username and password properties.

confluentSchemaRegistry:
- name: schema-registry
  tags: ["tag1"]
  version: 1    
  configuration:
    schemaRegistryUrls:
      value:
        - http://my-sr.host1:8081
        - http://my-sr.host2:8081
    username: 
      value: my-username
    password: 
      value: my-password

TLS with custom truststore

A custom truststore is needed when the Schema Registry is served over TLS (encryption-in-transit) and the Registry’s certificate is not signed by a trusted CA.

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslKeystore:
        fileRef:
          filePath: /path/to/my/keystore.jks
      sslKeystorePassword: 
        value: myPassword

Hard or soft delete

By default, Lenses will use hard delete for Schema Registry. To use soft delete, add the following property:

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      hardDelete:
        value: true      

AWS Glue

Some connections depend on others. One example is the AWS Glue Schema Registry connection, which depends on an AWS connection. These are examples of provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it.

Using AWS Access Key

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Access Key
      accessKeyId: 
        value: my-access-key-id
      secretAccessKey: 
        value: my-secret-access-key
      region: 
        value: eu-west-1
      
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: my-aws-connection
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
          value: arn:aws:glue:region:123123123:registry/my-registry

Using AWS Credentials Chain

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Credentials Chain
      region: 
        value: eu-west-1
        
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:123123123:registry/my-registry

PLAINTEXT

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

• kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.

• protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL 

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword

SASL_PLAINTEXT vs SASL_SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is if:

1. The transport layer is encyrpted (SSL)

2. The SASL mechanisn for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

Apart from that, when encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

Following are a few examples of SASL_PLAINTEXT and SASL_SSL

SASL_SSL

PLAIN

Encrypted communication and basic username and password for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";      

AWS_MSK_IAM

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

GSSAPI

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword  
    saslMechanism: 
      value: GSSAPI
    saslJaasConfig:
      value: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";      
     keytab:
       file: /path/to/keytab.jks

SASL_PLAINTEXT

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

SCRAM-SHA-256

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";      

SCRAM-SHA-512

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";    

Advanced Client Configuration

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configurationObject:
    kafkaBootstrapServers:
      value:
       - PLAINTEXT://your.kafka.broker.0:9092
    protocol: 
      value: PLAINTEXT
    additionalProperties:
      value:
        isolation.level: "read_committed"
        acks: "all"
        ssl.endpoint.identification.algorithm: "https"

Simple configuration, with JMX metrics

The URLs (workers) should always have a scheme defined (http:// or https://).

This example uses an optional AES-256 key. The key decodes values encoded with AES-256 to enable passing encrypted values to connectors. It is only needed if your cluster uses AES-256 Decryption plugin.

kafkaConnect:
  - name: my-connect-cluster-name
    version: 1    
    tags: ["tag1"]
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    

Basic authentication

For Basic Authentication, define username and password properties.

kafkaConnect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      username: 
        value: my-username
      password: 
        value: my-password

TLS with custom truststore

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

kafkaConnect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      sslTruststore:
        file: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

kafkaConnect:
  name: my-connect-cluster-name
  tags: ["tag1"]
  version: 1    
  configuration:
    workers:
      value:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083    
   sslKeystore:
      file: /path/to/my/keystore.jks
    sslKeystorePassword: 
      value: myPassword

The CLI allows you to import and export resources to and from files.

Import is done on a per-resource basis, the directory structure defined by the CLI. A base directory can be provided by the —dir flag.

Processors, connectors, topics, and schemas have an additional prefix flag to restrict resources to export.

#export
lenses-cli export acls --dir my-dir
lenses-cli export alert-channels --dir my-dir
lenses-cli export alert-settings --dir my-dir
lenses-cli export connections --dir my-dir
lenses-cli export connectors --dir my-dir
lenses-cli export processors --dir my-dir
lenses-cli export quota --dir my-dir
lenses-cli export schemas --dir my-dir
lenses-cli export topics --dir my-dir
lenses-cli export policies --dir my-dir
lenses-cli export groups --dir my-dir
lenses-cli export serviceaccounts --dir my-dir

#import
lenses-cli import acls --dir my-dir
lenses-cli import alert-channels --dir my-dir
lenses-cli import alert-settings --dir my-dir
lenses-cli import connections --dir my-dir
lenses-cli import connectors --dir my-dir
lenses-cli import processors --dir my-dir
lenses-cli import quota --dir my-dir
lenses-cli import schemas --dir my-dir
lenses-cli import topics --dir my-dir
lenses-cli import policies --dir my-dir
lenses-cli import groups --dir my-dir
lenses-cli import serviceaccounts --dir my-dir

Directory structure

The expected directory structure is:

my-dir
├── alert-settings
│   └── alert-setting.yaml
├── apps
│   ├── connectors
│   │   ├── connector-1.yaml
│   │   └── connector-2.yaml
│   └── sql
├── groups
│   └── groups.yaml
├── kafka
│   ├── quotas
│   │   └── quotas.yaml
│   └── topics
│       ├── topic-1.yaml
│       └── topic-2.yaml
├── policies
│   └── policies-city.yaml
├── service-accounts
│   └── service-accounts.yaml
└── schemas
    ├── schema-1.yaml
    └── schema-2.yaml

SQL Processors

Only the update of name, cluster name, namespace, and runner are allowed. Changes to the SQL are effectively the creation of a new Processor.

The Lenses docker image can be configured via environment variables or via volume mounts for the configuration files (lenses.conf, security.conf).

Running the Docker

docker run --name lenses \
  -e LENSES_PORT=3030\
  -e LENSES_SECURITY_USER=admin \
  -e LENSES_SECURITY_PASSWORD=sha256:8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 \
  -p 3030:3030\
  -p 9102:9102 \
   lensesio/lenses:latest

Open Lenses in your browser, log in with admin/admin and configure your brokers and add your license.

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses.conf and security.conf can be mounted directly as

• /mnt/settings/lenses.conf

• /mnt/secrets/security.conf

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

• /data/storage

• /data/plugins

• /data/logs

• /data/kafka-streams-state

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Lenses upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Lenses may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Lenses TLS and Global JVM Trust Store

By default, the Lenses serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

1. Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets

2. Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting Lenses.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the license, settings, and data) have the correct permission set.

The AWS Marketplace offering requires AWS MSK (Managed Apache Kafka) to be available. Optionally, AWS RDS (or any other PostgreSQL-compatible database) can be configured for Lenses to store its state.

The following AWS resources are created:

• An EC2 instance that runs Lenses;

• A SecurityGroup to allow network access to the Lenses UI;

• A SecurityGroupIngress for Lenses to connect to MSK;

• A CloudWatch LogGroup where Lenses stores its logs;

• An IAM Role to allow the EC2 instance to store logs;

• An IAM InstanceProfile to pass the role to the EC2 instance;

• Optionally if enabled during deployment: an IAM Policy to allow the EC2 instance to emit CloudWatch metrics.

Deployment takes approximately three minutes.

AWS Marketplace Installation

Select CloudFormation Template, Lenses EC2 and your region.

Choose Launch CloudFormation.

Continue with the default options for creating the stack in the AWS wizard.

Fill in the parameters at Specify stack details.

• Deployment Here the EC2 instance size and password for the Lenses admin user are set. A t2.large instance size is recommended;

• Network Configuration This section controls the network settings of the Lenses EC2 instance. The ingress allows access to the Lenses UI only from particular IP addresses;

• MSK Set the Security Group ID to that of your MSK cluster. A rule will be added to it so that Lenses can communicate with your cluster. You can find the ID by navigating in the AWS console to your MSK cluster and then under Properties -> Networking settings;

• Monitoring Optionally produce the Lenses logs to CloudWatch;

• Storage Lenses stores its state in a database locally on the EC2 instance’s disk or in a PostgreSQL database. Local storage is a development/quickstart option and is not suitable for production use. It is advised to use a Postgres database for smoother upgrades.

Review the stack.

Accept the terms and conditions and create the stack.

Once the stack has deployed, go to the Output tab and click on the FQDN link. If there are no outputs listed you might need to press the refresh button.

Login to Lenses with admin and the password value you have submitted for the parameter LensesAdminPassword.

IAM Support

Lenses supports connection to MSK brokers via IAM. If Lenses is deployed on an EC2 instance it will use the default credential chain loader to authenticate and connect to MSK.

Supported Regions

The following Regions are supported:

• us-east-1;

• us-east-2;

• us-west-1;

• us-west-2;

• ca-central-1;

• eu-central-1;

• eu-west-1;

• eu-west-2;

• eu-west-3;

• ap-southeast-1;

• ap-southeast-2;

• ap-south-1;

• ap-northeast-1;

• ap-northeast-2;

• sa-east-1.

Security Recommendations

Please:

• Do not use your AWS root user for deployment or operations;

• Follow the least privileges principle when granting access to individual IAM user accounts;

• Avoid allowing traffic to the Lenses UI from a broad CIDR block where a more specific block could be used.

Pricing

AWS billing applies for the EC2 instance, CloudWatch logs and optionally CloudWatch metrics.

For the hourly billed version additional hourly charges apply, which depend on the instance size. For the Bring Your Own License (BYOL) you can get a free trial license here.

Troubleshooting

In case you run into problems, e.g. you cannot connect to Lenses, then the logs could provide more information. The easiest route to do this is to go to CloudWatch in the AWS console. Here, find the log group corresponding to your deployment (it has the same name as the deployment) and pick a log stream. The stream with the /lenses.log suffix contains all log lines regardless of the log level; the stream with the /lenses-warn.log suffix only contains warning-level logs.

If the above fails, for example, because the logs integration is broken, you can SSH into the EC2 instance. Lenses is installed into /opt/lenses, the logs can be found under /opt/lenses/logs for further inspection

Provisioning

A dedicated API, called provisioning, is available to handle bootstrapping key connections at installation time. This allows you to fully install and configure key connections such as Kafka, Schema Registry, Kafka Connect, and Zookeepers in one go. You can use either of the following approaches depending on your needs:

Defining a Connection

Connections are defined in theprovisioning.yaml. This file is divided into components, each component representing a type of connection.

Each component must have:

1. Name - This is the free name of the connection

2. Version set to 1

3. Optional tags

4. Configuration - This is a list of keys/values and is dependent on the component type.

For a full list of configuration options for the connect see Provisioning API Spec.

First, add the Helm Chart repository using the Helm command line:

helm repo add lensesio https://helm.repo.lenses.io
helm repo update

Use helm to install Lenses with default values:

helm install lenses lensesio/lenses --namespace lenses --create-namespace

The default install of Lenses will place Lenses in bootstrap mode, you can add the connections to Kafka manually and upload your license or automation with provisioning. Please refer to the GitHub values.yaml for all options.

Provisioning

To automatically provision the connections to Kafka and other systems set the .Values.lenses.provision.connections to be the YAML definition of your connections. For a full list of the connection types supported see Provisioning.

The chart will render the full YAML specified under this setting as the provisioning.yaml file.

Alternatively you can use a second YAML file, which contains only the connections pass them at the command line when installing:

helm install lenses \
charts/lenses \
--values charts/lenses/values.yaml \
--values provisioning.yaml

Helm Chart components

The chart uses:

1. Secrets to store Lenses Postgres credentials and authentication credentials

2. Secrets to store connection credentials such as Kafka SASL_SCRAM password or password for SSL JKS stores.

3. Secrets to hold the base64 encoded values of the JKS stores

4. ConfigMap for Lenses configuration overrides

5. Cluster roles and role bindings (optional).

Secrets and config maps are mounted as files under the mount /mnt:

1. settings - holds the lenses.conf

2. secrets - holds the secrets Lenses and license

3. provision-secrets - holds the secrets for connections in the provisioning.yaml file

4. provision-secrets/files - holds any file needed for a connection, e.g. JKS files.

Cluster RBAC

The Helm chart creates Cluster roles and bindings, these are used by SQL Processors if the deployment mode is set to KUBERENTES. They are used so that Lenses can deploy and monitor SQL Processor deployments in namespaces.

To disable the RBAC set: rbacEnabled: false

If you want to limit the permissions Lenses has against your Kubernetes cluster, you can use Role/RoleBinging resources instead.

To achieve this you need to create a Role and a RoleBinding resource in the namespace you want the processors deployed to.

For example:

• Lenses namespace = lenses-ns

• Processor namespace = lenses-proc-ns

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role
  namespace: lenses-proc-ns
rules:
- apiGroups: [""]
  resources:
    - namespaces
    - persistentvolumes
    - persistentvolumeclaims
    - pods/log
  verbs:
    - list
    - watch
    - get
    - create
- apiGroups: ["", "extensions", "apps"]
  resources:
    - pods
    - replicasets
    - deployments
    - ingresses
    - secrets
    - statefulsets
    - services
  verbs:
    - list
    - watch
    - get
    - update
    - create
    - delete
    - patch
- apiGroups: [""]
  resources:
    - events
  verbs:
    - list
    - watch
    - get
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role-binding
  namespace: lenses-proc-ns
subjects:
- kind: ServiceAccount
  namespace: lenses-ns
  name: default
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io

Finally you need to define in Lenses configuration which namespaces can Lenses access. To achieve this amend values.yaml to contain the following:

lenses:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "lenses-processors"
        ]
      }      

lenses.conf

The main configurable options for lenses.conf are available in the values.yaml under the lenses object. These include:

• Authentication

• Database connections

• SQL processor configurations

To apply other static configurations use lenses.append.conf, for example:

lenses:
  append:
    conf: |
      lenses.interval.user.session.refresh=40000

Authentication secrets

Set accordingly under**lenses.security.**

lenses:
  security:
    defaultUser:
      username: admin
      password: admin

For SSO set lenses.security.saml

lenses:
  security:
    saml:
      enabled: true
      baseUrl: "https://lenses-prod.eastus2.cloudapp.azure.com"
      provider: "azure"
      keyStoreFileData: |-
                somebase64encodedvalue
      keyStorePassword: "password"
      keyPassword: "password"
      metadataFileData: |-
                somebase64encodedvalue=

Postgres

To use Postgres as the backing store for Lenses set the details in the lenses.storage.postgres object.

storage:
    postgres:
      enabled: false
      host:
      port: 
      username:
      password:
      database:
      schema:        

If Postgres is not enabled a default embedded H2 database is used. To enable persistence for this data:

persistence:
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 5Gi

External Secrets

The chart relies on secrets for sensitive information such as Passwords. Secrets can rotate and are commonly stored in an external store such as Azure KeyVault, Hashicorp Vault or AWS Secrets Manager.

If you wish to have the chart use external secrets that are synchronized with these providers, set the following for the Lenses user:

  security:
    defaultUser:  
      # username: external # "external" that tells Lenses to look for a Secret
      # password: external # Same here.
      # usernameSecretKeyRef:
      #   name: my-existing-secret
      #   key: the-username-key
      # passwordSecretKeyRef:
      #   name: my-existing-secret
      #   key: the-password-key

For Postgres, add additional ENV variables via the lenses.additionalEnv object to point to your secret and set the username and password to external in the Postgres section.

      # - name: LENSES_STORAGE_POSTGRES_PASSWORD
      #   valueFrom:
      #     secretKeyRef:
      #       name: [SECRET_RESOURCE_NAME]
      #       key: [SECRET_RESOURCE_KEY]

Ingress & Services

Ingress and service resources are supported.

Enabled an Ingress resource in the values.yaml:

ingress:
  enabled: false
  host:
  annotations: {}
  tls:
    enabled: false
    crt: |-
    key: |-

Enable a service resource in the values.yaml:

# Lenses service
service:
  enabled: true
  type: ClusterIP
  annotations: {}
  externalTrafficPolicy:
  loadBalancerIP: 130.211.x.x
  loadBalancerSourceRanges:
    - 0.0.0.0/0

Controlling resources

To control the resources used by Lenses:

# Resource management
resources:
  requests:
    cpu: 1
    memory: 4Gi
  limits:
    cpu: 2
    memory: 5Gi

Enabling SQL Processors in K8s mode

To enable SQL processor in KUBERENTES mode and control the defaults:

sql:
    # processorImage: eu.gcr.io/lenses-container-registry/lenses-sql-processor
    # processorImageTag: 2.3
    mode: IN_PROC
    heap: 1024M
    minHeap: 128M
    memLimit: 1152M
    memRequest: 128M
    livenessInitialDelay: 60 seconds

Prometheus metrics

Prometheus metrics are automatically exposed on port 9102 under /metrics.

Example Values files

For Connections, see Provisioning examples. You can also find examples in the Helm chart repo.

Building on the provisioning.yaml, API provisiong allows for uploading the files directly Lenses from anywhere with network access and without access to the host where Lenses is installed.

Uploading supporting files

Many connections need files, for example, to secure Kafka with SSL you will need a keystore and optionally a trust store.

To reference a file in the, for the configuration option set the key to be "file" and the value to reference in the API request. For example, given:

    configuration:
      protocol:
        value: SASL_SSL
      sslKeystore:
        file: "my-keystore-file"

To upload the file to be used for the configuration option sslKeystore: add the following to the request:

--form "my-keystore-file=@${PATH_TO_KEYSTORE_FILE};type=application/octet-stream" \

API Call

1. Set the type to application/octet-stream.

2. The name of the part in the multipart request (supporting files) should match the value of the property pointing to the mounted file in the provisioning.yaml descriptor. This ensures accurate mapping and referencing of files.

3. Set LENSES_SESSION_TOKEN as the value of the Lenses Service Account token you want to use to automate provisioning.

curl --location --request PUT "${LENSES_ENV}/api/v1/state/connections" \
   --header "X-Kafka-Lenses-Token: ${LENSES_SESSION_TOKEN}" \
   --header 'Content-Type: multipart/form-data' \
   --header 'Content-Disposition: form-data;' \
   --form "my-keystore-file=@${PATH_TO_KEYSTORE_FILE};type=application/octet-stream" \
   --form 'provisioning=@"resources/provisioning.yaml";type=text/plain(utf-8)' 

In this example, the provisioning.yaml is read from provisioning=@"resources/provisioning.yaml.

The provisioning.yaml contains a reference to "my-keystore-file" which is loaded from @${PATH_TO_KEYSTORE_FILE};type=application/octet-stream

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime. i.e. inject an environment variable from GitHub secrets for passwords.

sslKeystorePassword:
  value: ${ENV_VAR_NAME}

Connections are defined in the provisioning.yaml file. Lenses will then watch the file and resolve the desired state, applying connections defined in the file.

Enabling File Watcher Provisioning

File watcher provisioning must be explicitly enabled. Set the following in the lenses.conf file:

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir
# The interval at which Lenses will check for updates 
# in the file seconds
lenses.provisioning.interval=10s

Directory layout

Lenses expects a set of files in the directory, defined by lenses.provisioning.path. The structure of the directory must follow:

1. files/ directory for storing any certificates, JKS files or other files needed by the connection

2. provisioning.yaml - This is the main file, holding the definition of the connections

3. license.json - Your lenses license file

➜  ~ tree provisioning-folder
provisioning-folder
├── files
│   └── tuststore.jks
├── license.json
└── provisioning.yaml

4 directories, 0 files
➜  ~

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm, the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime.

sslKeystorePassword:
  value: ${ENV_VAR_NAME}

Referencing files

Many connections need files, for example, to secure Kafka with SSL you will need a key store and optionally a trust store.

To reference a file in the provisioning.yaml, for example, given:

    configuration:
      protocol:
        value: SASL_SSL
      sslKeystore:
        file: "my-keystore.jks"

a file called my-keystore.jks is expected in the files directory. This file will be used for the key store location.

What’s in the Box?

Lenses Box contains all components of the Apache Kafka ecosystem, CLI tools, and synthetic data streams.

Starting the Box

1. To start with the Box get your free development license online.

2. Install and run the Docker

docker run --rm \
    -p 3030:3030 \
    --name=dev \
    --net=host \
    -e EULA="https://dl.lenses.stream/d/?id=CHECK_YOUR_EMAIL_FOR_KEY" \
    lensesio/box   

Open Lenses in your browser, log in with admin/admin.

Kafka Docker advertisement

The broker in the Kafka docker has a broker id 101 and advertises the listener configuration endpoint to accept client connections.

If you run Docker on macOS or Windows, you may need to find the address of the VM running Docker and export it as the advertised listener address for the broker (On macOS it usually is 192.168.99.100). At the same time, you should give the lensesio/box image access to the VM’s network:

docker run -e EULA="CHECK_YOUR_EMAIL_FOR_KEY" \
           -e ADV_HOST="192.168.99.100" \
           --net=host --name=dev \
           lensesio/box

If you run on Linux, you don’t have to set the ADV_HOST , but you can do something cool with it. If you set it to be your machine’s IP address, you can access Kafka from any clients in your network.

If you decide to run a box in the cloud, you (and all your team) can access Kafka from your development machines. Remember to provide the public IP of your server as the kafka advertised host for your producers and consumers to access it.

Kafka Docker JMX

Kafka JMX metrics are enabled by default. Refer to ports once you expose the relevant port, ie. -p 9581:9581 you can connect to JMX with

jconsole localhost:9581

Custom hostname

If you are using docker-machine or setting this up in a Cloud or DOCKER_HOST is a custom IP address such as 192.168.99.100, you will need to use the parameters --net=host -e ADV_HOST=192.168.99.100.

docker run --rm \
    -p 3030:3030 \
    --net=host \
    -e ADV_HOST=192.168.99.100 \
    -e EULA="https://dl.lenses.stream/d/?id=CHECK_YOUR_EMAIL_FOR_KEY" \
    lensesio/box    

Docker data persistence

To persist the Kafka data between multiple executions, provide a name for your Docker instance and do not set the container to be removed automatically (--rm flag). For example:

docker run \
    -p 3030:3030 -e EULA="CHECK_YOUR_EMAIL_FOR_KEY" \
    --name=dev lensesio/box

Once you want to free up resources, just press Control-C. Now you have two options: either remove the Docker:

docker rm dev

Or use it at a later time and continue from where you left off:

docker start -a devhshel

Port Numbers

Advanced options

FAQ

How can I run offline?

Download your key locally and run the command:

LFILE=`cat license.json`
docker run --rm -it -p 3030:3030 -e LICENSE="$LFILE" lensesio/box:latest

How much memory to allocate?

The container is running multiple services, and it is recommended to allocate 5GB of RAM to the docker (although it can operate with even less than 4GB).

To reduce the memory footprint, it is possible to disable some connectors and shrink the Kafka Connect heap size by applying these options (choose connectors to keep) to the docker run command:

-e DISABLE=azure-documentdb,cassandra,elastic5,ftp,influxdb,jms,mongodb,mqtt,redis
-e CONNECT_HEAP=512m