Docker

The official Lenses docker image is available at the docker hub under landoop/lenses. The build is automated and the source code is available at GitHub.

Lenses docker image can be used instead of the Lenses archive or Lenses helm chart —for which it serves as the base.

The Lenses Box docker image, please check the Lenses Box section.

Lenses Configuration

The image uses the standard practice of converting environment variables to configuration options for Lenses. The convention is that letters are uppercase in environment variables and lowercase in Lenses configuration options, whilst underscores in environment variables translate to dots in configuration options. Only options starting with LENSES_ get processed.

Some examples include:

• Configuration option lenses.port would be set via environment variable LENSES_PORT
• Configuration option lenses.schema.registry.urls would be set via environment variable LENSES_SCHEMA_REGISTRY_URLS

Necessary configuration options are:

• LENSES_PORT
• LENSES_KAFKA_BROKERS
• LENSES_ZOOKEEPER_HOSTS
• LENSES_SECURITY_MODE
• LENSES_SECURITY_GROUPS
• LENSES_SECURITY_USERS

If LDAP is used instead of the basic security mode, then LENSES_SECURITY_USERS should be replaced with the options for LDAP setup.

Other important configuration options include:

• LENSES_SCHEMA_REGISTRY_URLS
• LENSES_CONNECT_CLUSTERS

More information may be found in the configuration section.

Optionally separate settings can be mounted as volumes under /mnt/settings or /mnt/secrets. Mounting a whole Lenses configuration file or a safety valve to be appended to the auto-generated one, is also supported. For more information about these methods and various quirks of environment variable based configuration as well as secret management, please continue reading below.

License File

Lenses needs your license file in order to work. If you don’t have one you may request a trial license or contact us for further information.

The license file may be provided to the docker image via three methods:

1. As a file, mounted at /license.json or /mnt/secrets/license.json
2. As the contents of the environment variable LICENSE
3. As a URL resource that will download on container startup via LICENSE_URL

Volumes

Lenses stores its data within Kafka, so docker volumes can be avoided if desired.

The docker image exposes two volumes, /data/logs and /data/kafka-streams-state which can be used if desired.

The former (/data/logs) is where Lenses logs are stored. The software also logs into stdout, so your existing log management solutions can be used.

The latter (/data/kafka-streams-state) is created when using LSQL in IN_PROC mode, that is LSQL queries run within Lenses. In such case, Lenses takes advantage of this scratch directory to cache LSQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so Lenses won’t have to rebuild the cache during a restart.

Process UID/GID

Lenses docker does not require running as root.

The default user in the image is set to root for convenience. Upon start, the initialization script will use the root privileges to make sure all directories and files have the correct permissions, then drop to user nobody and group nogroup (65534:65534) before starting Lenses.

If the image is started without root privileges, Lenses will start successfully under the effective uid:gid applied. In such case and if volumes are used —for the license, settings or data—, it is the responsibility of the operator to make sure that Lenses have permission to access these.

Broker Authentication

If the Kafka Cluster uses authentication, some additional files are needed in order to setup Lenses —or any Kafka client for that matter.

For SSL a truststore and a keystore may be needed, whilst for SASL/GSSAPI a jaas configuration file, a keytab and the system-wide kerberos configuration file (krb5.conf) are needed.

Whilst a manual setup is possible, the docker image offers the ability to provide the content for extra files via secrets, mounted as files or as environment variables.

The entries below are expected to be under any of the following: /mnt/files, /mnt/secrets, /run/secrets or exposed as environment variables. They should have the content of each file, either as is or encoded as base64. The docker image will decode them automatically if needed.

FILECONTENT_SSL_KEYSTORE

The SSL/TLS keystore to use for connecting to brokers that require or request authentication via SSL. The format should be binary (as is) or encoded to base64. If set, the startup script will automatically set lenses.kafka.settings.consumer.ssl.keystore.location and lenses.kafka.settings.producer.ssl.keystore.location to the proper path, so they should be not set.

FILECONTENT_SSL_TRUSTSTORE

The SSL/TLS truststore to use for verifying SSL connections to the brokers. The format should be binary (as is) or encoded to base64. If set, the startup script will automatically set lenses.kafka.settings.consumer.ssl.truststore.location and lenses.kafka.settings.producer.ssl.truststore.location to the proper path, so they should be not set.

FILECONTENT_KEYTAB

The Kerberos keytab to use for authenticating with a KDC in order to connect to brokers that require SASL/GSSAPI authentication. The format should be binary (as is) or encoded to base64. The keytab will be decoded if needed and copied to /data/keytab.

FILECONTENT_JAAS

The JAAS Login Configuration File for setting the Kerberos (SASL/GSSAPI) authentication to the brokers and optionally to zookeeper. The format should be text (as is) or encoded to base64. Please note that if the keytab is added via FILECONTENT_KEYTAB, then its location will be /data/keytab. If set, the startup script will add automatically to LENSES_OPTS the java.security.auth.login.config option and point it to the proper location, so it should not be set. Find out more about jaas.conf at SASL/GSSAPI (Kerberos) Authentication.

FILECONTENT_KRB5

The Kerberos Configuration file (krb5.conf) that contains the configuration for Kerberos clients, such as KDC location, realms, defaults. The format should be text (as is) or encoded to base64. If set, the startup script will add automatically to LENSES_OPTS the java.security.krb5.conf option and point it to the proper location, so it should not be set.

For more information about using Lenses with a security enabled Kafka cluster, please refer the relevant documentation sections: SSL Authentication and Encryption, SASL/GSSAPI (Kerberos) Authentication, SASL_SSL Authentication and Encryption, SASL/SCRAM Authentication.

Configuration via Files and Secrets

Lenses configuration options can be mounted as files under /mnt/settings and /mnt/secrets. The latter is usually used for options that contain secrets, such as LENSES_SECURITY_USERS and in conjunction with the underlying container orchestrator system’s secret management —such as kubernetes secrets.

For this functionality, a file with the name of the option’s environment variable and content its value must be used. As an example, to set lenses.port=9991, one would mount a file under /mnt/settings/LENSES_PORT with content 9991.

If the traditional configuration approach with files lenses.conf and security.conf is desired instead, they should be mounted under either directory, such as /mnt/settings/lenses.conf and /mnt/secrets/security.conf. Special care is needed for options lenses.secret.file and lenses.license.file which point to the license and secrets configuration files. It is advised to omit them and the initialization script will take care to append them correctly to the provided configuration files. If not omited, it is the responsibility of the operator to set them correctly under the paths these files are mounted. When the traditional configuration files are used, environment variables are not processed.

A hybrid approach, mixing configuration via environment variables but also files is supported as well via the files /mnt/settings/lenses.append.conf and /mnt/settings/security.append.conf. The contents of these files will be appended to lenses.conf and security.conf respectively after the environment variables are processed —and thus take priority.

JAVA Settings

Java and JVM settings may be set as described in java options configuration section. The most commonly used setting is LENSES_HEAP_OPTS which restricts the memory usage of Lenses. The default value is -Xmx3g -Xms512m which permits Lenses to use as much as 3GB of memory for heap space.

Configuration Quirks

Lenses configuration is in HOCON format which at times can be challenging to convert to from other formats, such as configuration via environment variables, especially when these are set via nonstandard channels, such as yaml and docker environment files.

Yaml is well supported and multiline variables are supported. Quotes should be avoided unless they are needed as literals, like in the url and jmx sections of LENSES_ZOOKEEPER_HOSTS. An example follows:

environment:
  LENSES_PORT: 9991
  LENSES_KAFKA_BROKERS: PLAINTEXT://kafka-1:9092,PLAINTEXT://kafka-2:9092,PLAINTEXT://kafka-3:9092
  LENSES_ZOOKEEPER_HOSTS: |
      [
        {url:"zookeeper-1:2181",jmx:"zookeeper-1:9585"},
        {url:"zookeeper-2:2181",jmx:"zookeeper-2:9585"},
        {url:"zookeeper-3:2181",jmx:"zookeeper-3:9585"}
      ]
  LENSES_SCHEMA_REGISTRY_URLS: |
    [
      {url:"http://registry-1:8081",jmx:"registry-1:9582"},
      {url:"http://registry-2:8081",jmx:"registry-2:9582"}
    ]
  LENSES_CONNECT_CLUSTERS: |
    [
      {name:"production",
      urls: [
              {url:"http://connect-1:8083",jmx:"connect-1:9584"},
              {url:"http://connect-2:8083",jmx:"connect-2:9584"}
              {url:"http://connect-3:8083",jmx:"connect-3:9584"}
            ],
      statuses:"connect-statuses",
      configs:"connect-configs",
      offsets:"connect-offsets"}
    ]
  LENSES_ALERT_PLUGINS_SLACK_ENABLED: "false"
  LENSES_SECURITY_MODE: BASIC
  LENSES_SECURITY_GROUPS: |
      [
        {"name": "adminGroup", "roles": ["admin", "write", "read"]},
        {"name": "writeGroup", "roles": ["read", "write"]},
        {"name": "readGroup",  "roles": ["read"]},
        {"name": "nodataGroup",  "roles": ["nodata"]}
      ]
  LENSES_SECURITY_USERS: |
      [
        {"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
        {"username": "write", "password": "write", "displayname": "Write User", "groups": ["writeGroup"]},
        {"username": "read", "password": "read", "displayname": "Read Only", "groups": ["readGroup"]},
        {"username": "nodata", "password": "nodata", "displayname": "No Data", "groups": ["nodataGroup"]}
      ]

Docker environment files do not support multiline entries. Again, quotes should be used only when literals are expected and avoided in any other case.

LENSES_PORT=9991
LENSES_KAFKA_BROKERS=PLAINTEXT://kafka-1:9092,PLAINTEXT://kafka-3:9092,PLAINTEXT://kafka-3:9092
LENSES_ZOOKEEPER_HOSTS=[{url:"zookeeper-1:2181",jmx:"zookeeper-1:9585"},{url:"zookeeper-2:2181",jmx:"zookeeper-2:9585"},{url:"zookeeper-3:2181",jmx:"zookeeper-3:9585"}]
LENSES_SCHEMA_REGISTRY_URLS=[{url:"http://registry-1:8081",jmx:"registry-1:9582"},{url:"http://registry-2:8081",jmx:"registry-2:9582"}]
SD_CONFIG=provider=gce zone_pattern=europe-west1.*
SD_CONNECT_NAMES=production
SD_CONNECT_JMX_PORTS=9584
SD_CONNECT_FILTERS=tag_value=connect-worker
LENSES_SECURITY_GROUPS=[{"name": "adminGroup", "roles": ["admin", "write", "read"]}, {"name": "writeGroup", "roles": ["read", "write"]}]
LENSES_SECURITY_USERS=[{"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]}, {"username": "write", "password": "write", "displayname": "Write User", "groups": ["writeGroup"]}]

Monitoring and Prometheus

Lenses use the JVM, as such they can expose a JMX endpoint where applications can connect to access metrics. To enable the JMX endpoint, please set the environment variable LENSES_JMX_PORT. Depending on your enviroment, additional settings may be needed, such as:

LENSES_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=[LENSES_JMX_HOST] -Dcom.sun.management.jmxremote.rmi.port=[LENSES_JMX_PORT]"

A Prometheus endpoint is provided by default, through a jmx_exporter instance that is loaded as agent into Lenses. Its port is 9102 and cannot be altered —but may be exposed under a different port as per docker settings. The java agent is loaded always. It exposes process and kafka client metrics.

It is a common practice when deploying into Kubernetes to expose a liveness endpoint. Lenses docker image does not have a dedicated endpoint but the address of Lenses itself can be used for this purpose.

Custom Serde

To add custom serde to the docker image, either mount the serde jars directory under /opt/lenses/serde:

docker run -v /path/to/serde/jar/directory:/opt/lenses/serde ...

Or mount it under a custom directory and export the LENSES_SERDE_CLASSPATH_OPTS variable:

docker run -v /path/to/serde/jar/directory:/path/to/serde \
       -e LENSES_SERDE_CLASSPATH_OPTS=/path/to/serde \
       ...

Cloud Service Discovery

In version 2.0 service discovery came into the Lenses docker as a preview feature.

Find out about cloud service discovery here.

Examples

These examples serve more as a quick reference guide.

Docker-compose example configuration.

version: '2'
services:
  lenses:
    image: landoop/lenses:2.1
    environment:
      LENSES_PORT: 9991
      LENSES_KAFKA_BROKERS: "PLAINTEXT://broker.1.url:9092,PLAINTEXT://broker.2.url:9092"
      LENSES_ZOOKEEPER_HOSTS: |
        [
          {url:"zookeeper.1.url:2181", jmx:"zookeeper.1.url:9585"},
          {url:"zookeeper.2.url:2181", jmx:"zookeeper.2.url:9585"}
        ]
      LENSES_SCHEMA_REGISTRY_URLS: |
        [
          {url:"http://schema.registry.1.url:8081",jmx:"schema.registry.1.url:9582"},
          {url:"http://schema.registry.2.url:8081",jmx:"schema.registry.2.url:9582"}
        ]
      LENSES_CONNECT_CLUSTERS: |
        [
          {
            name:"data_science",
            urls: [
              {url:"http://connect.worker.1.url:8083",jmx:"connect.worker.1.url:9584"},
              {url:"http://connect.worker.2.url:8083",jmx:"connect.worker.2.url:9584"}
            ],
            statuses:"connect-statuses-cluster-a",
            configs:"connect-configs-cluster-a",
            offsets:"connect-offsets-cluster-a"
          }
        ]
      LENSES_SECURITY_MODE: BASIC
      # Secrets can also be passed as files. Check _examples/
      LENSES_SECURITY_GROUPS: |
        [
          {"name": "adminGroup", "roles": ["admin", "write", "read"]},
          {"name": "readGroup",  "roles": ["read"]}
        ]
      LENSES_SECURITY_USERS: |
        [
          {"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
          {"username": "read", "password": "read", "displayname": "Read Only", "groups": ["readGroup"]}
        ]
    ports:
      - 9991:9991
      - 9102:9102
    volumes:
      - ./license.json:/license.json
    network_mode: host

A Kubernetes pod and service example is available at here. More information about running Lenses inside Kubernetes, is available at the kubernetes and helm section.