Docker¶

The official Lenses docker image is available at the docker hub under landoop/lenses. The build is automated and the source code is available at GitHub.

Lenses docker image can be used instead of the Lenses archive or Lenses helm chart —for which it serves as the base.

The Lenses Box docker image, please check the Lenses Box section.

Lenses Configuration¶

The image uses the standard practice of converting environment variables to configuration options for Lenses. The convention is that letters are uppercase in environment variables and lowercase in Lenses configuration options, whilst underscores in environment variables translate to dots in configuration options. Only options starting with LENSES_ get processed.

Some examples include:

Configuration option lenses.port would be set via an environment variable LENSES_PORT
Configuration option lenses.schema.registry.urls would be set via environment variable LENSES_SCHEMA_REGISTRY_URLS

Necessary configuration options are:

LENSES_KAFKA_BROKERS
LENSES_SECURITY_MODE
LENSES_SECURITY_GROUPS
LENSES_SECURITY_USERS
license file

If LDAP is used instead of the basic security mode, then LENSES_SECURITY_USERS should be replaced with the options for LDAP setup.

Other important configuration options for an optimal setup may include:

LENSES_SCHEMA_REGISTRY_URLS
LENSES_CONNECT_CLUSTERS
LENSES_ZOOKEEPER_HOSTS

More information may be found in the configuration section.

Optionally separate settings can be mounted as volumes under /mnt/settings or /mnt/secrets. Mounting a whole Lenses configuration file or a safety valve to be appended to the auto-generated one, is also supported. For more information about these methods and various quirks of environment variable based configuration as well as secret management, please continue reading below.

License File¶

Lenses needs your license file in order to work. If you don’t have one you may request a trial license or contact us for further information.

The license file may be provided to the docker image via three methods:

As a file, mounted at /license.json or /mnt/secrets/license.json
As the contents of the environment variable LICENSE
As a URL resource that will download on container startup via LICENSE_URL

Volumes¶

Lenses stores some of its data in Kafka and some under the /data/storage docker volume. Also, certain ephemeral data and plugins may be stored on volumes. Depending on your requirements you may need to set up persistent storage for the container.

The docker image exposes four volumes in total, where cache, logs, plugins and persistent data are stored: /data/storage, /data/plugins, /data/logs and /data/kafka-streams-state.

Storage volume: Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between docker runs and/or Lenses upgrades, the volume must be managed externally (persistent volume).
Plugins volume: Resides under /data/plugins it’s where classes that extend Lenses may be added —such as custom serde, LDAP filters, UDFs for the Lenses SQL table engine and custom_http implementations. Learn more about plugins.
Logs volume: Resides under /data/logs, logs are stored here. The application also logs to stdout, so for most cases, the log files aren’t needed. Learn more about logs.
KStreams state volume: Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC mode —that is when Lenses SQL processors run within the Lenses process. In such a case, Lenses takes advantage of this scratch directory to cache L.SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

Note

Currently under the storage volume, only Data Policies are stored. If you don’t use this functionality, you can omit to set a persistent volume for it. You will have to keep your eye on upgrade notes though, as in the future more data may be stored there.

Process UID/GID¶

Lenses docker does not require running as root.

The default user in the image is set to root for convenience. Upon start, the initialization script will use the root privileges to make sure all directories and files have the correct permissions, then drop to user nobody and group nogroup (65534:65534) before starting Lenses.

If the image is started without root privileges, Lenses will start successfully under the effective uid:gid applied. In such case and if volumes are used —for the license, settings or data—, it is the responsibility of the operator to make sure that Lenses have permission to access these.

Broker Authentication¶

If the Kafka Cluster uses authentication, some additional files are needed in order to setup Lenses —or any Kafka client for that matter. For SSL a truststore and a keystore may be needed whilst for SASL/GSSAPI a jaas configuration file, a keytab and the system-wide Kerberos configuration file (krb5.conf) are needed.

Although a manual setup is possible, the docker image offers the ability to provide the content for extra files via secrets, mounted as files or as environment variables. Especially for SSL, the docker supports SSL/TLS keys and certificates in both PEM and Java Keystore (JKS) formats.

The entries below are expected to be under any of the following: /mnt/files, /mnt/secrets, /run/secrets or exposed as environment variables. They should have the content of each file, either as is or encoded as base64. The docker image will decode them automatically if needed.

FILECONTENT_SSL_KEYSTORE: The SSL/TLS keystore to use for connecting to brokers that require or request authentication via SSL. The format should be binary (as is) or encoded to base64. If set, the startup script will automatically set lenses.kafka.settings.consumer.ssl.keystore.location and lenses.kafka.settings.producer.ssl.keystore.location to the proper path, so they should be not set. The location of the final keystore is /data/keystore.jks.
FILECONTENT_SSL_TRUSTSTORE: The SSL/TLS truststore to use for verifying SSL connections to the brokers. The format should be binary (as is) or encoded to base64. If set, the startup script will automatically set lenses.kafka.settings.consumer.ssl.truststore.location and lenses.kafka.settings.producer.ssl.truststore.location to the proper path, so they should be not set. The location of the final truststore is /data/truststore.jks.
FILECONTENT_SSL_KEY_PEM: The client SSL key in PEM format to use for connecting to brokers that require or request authentication via SSL. It should be used in conjuction with FILECONTENT_SSL_CERT_PEM. If set, the startup script will convert it (alongside the client SSL certificate) to a keystore file and automatically set lenses.kafka.settings.consumer.ssl.keystore.location, lenses.kafka.settings.consumer.ssl.keystore.password, lenses.kafka.settings.consumer.ssl.key.password, lenses.kafka.settings.producer.ssl.keystore.location, lenses.kafka.settings.producer.ssl.keystore.password, and lenses.kafka.settings.producer.ssl.key.password to the proper values, so they should be not set. The location of the final keystore is /data/keystore.jks.
FILECONTENT_SSL_CERT_PEM: The client SSL certificate in PEM format to use for connecting to brokers that require or request authentication via SSL. It should be used in conjuction with FILECONTENT_SSL_KEY_PEM. If set, the startup script will convert it (alongside the client SSL key) to a keystore file and automatically set lenses.kafka.settings.consumer.ssl.keystore.location, lenses.kafka.settings.consumer.ssl.keystore.password, lenses.kafka.settings.consumer.ssl.key.password, lenses.kafka.settings.producer.ssl.keystore.location, lenses.kafka.settings.producer.ssl.keystore.password, and lenses.kafka.settings.producer.ssl.key.password to the proper values, so they should be not set. The location of the final keystore is /data/keystore.jks.
FILECONTENT_SSL_CACERT_PEM: The CA certificate in PEM format to use for verifying SSL connections to the brokers. If set, thee startup script will convert it to a truststore file and automatically set lenses.kafka.settings.consumer.ssl.truststore.location, lenses.kafka.settings.consumer.ssl.truststore.password, lenses.kafka.settings.producer.ssl.truststore.location, and lenses.kafka.settings.producer.ssl.truststore.password to the proper values, so they should be not set. The location of the final truststore is /data/truststore.jks.
FILECONTENT_KEYTAB: The Kerberos keytab to use for authenticating with a KDC in order to connect to brokers that require SASL/GSSAPI authentication. The format should be binary (as is) or encoded to base64. The keytab will be decoded if needed and copied to /data/keytab.
FILECONTENT_JAAS: The JAAS Login Configuration File for setting the Kerberos (SASL/GSSAPI) authentication to the brokers and optionally to zookeeper. The format should be text (as is) or encoded to base64. Please note that if the keytab is added via FILECONTENT_KEYTAB, then its location will be /data/keytab. If set, the startup script will add automatically to LENSES_OPTS the java.security.auth.login.config option and point it to the proper location, so it should not be set. Find out more about jaas.conf at SASL/GSSAPI.
FILECONTENT_KRB5: The Kerberos Configuration file (krb5.conf) that contains the configuration for Kerberos clients, such as KDC location, realms, defaults. The format should be text (as is) or encoded to base64. If set, the startup script will add automatically to LENSES_OPTS the java.security.krb5.conf option and point it to the proper location, so it should not be set.

For more information about using Lenses with a security enabled Kafka cluster, please refer the relevant documentation sections: SSL, SASL/GSSAPI, SASL_SSL, SASL/SCRAM.

Configuration via Files and Secrets¶

Lenses configuration options can be mounted as files under /mnt/settings and /mnt/secrets. The latter is usually used for options that contain secrets, such as LENSES_SECURITY_USERS and in conjunction with the underlying container orchestrator system’s secret management —such as Kubernetes secrets.

For this functionality, a file with the name of the option’s environment variable and content its value must be used. As an example, to set lenses.port=9991, one would mount a file under /mnt/settings/LENSES_PORT with content 9991.

If the traditional configuration approach with files lenses.conf and security.conf is desired instead, they should be mounted under either directory, such as /mnt/settings/lenses.conf and /mnt/secrets/security.conf. Special care is needed for options lenses.secret.file and lenses.license.file which point to the license and secrets configuration files. It is advised to omit them and the initialization script will take care to append them correctly to the provided configuration files. If not omitted, it is the responsibility of the operator to set them correctly under the paths these files are mounted. When the traditional configuration files are used, environment variables are not processed.

A hybrid approach, mixing configuration via environment variables but also files is supported as well via the files /mnt/settings/lenses.append.conf and /mnt/settings/security.append.conf. The contents of these files will be appended to lenses.conf and security.conf respectively after the environment variables are processed —and thus take priority.

JAVA Settings¶

Java and JVM settings may be set as described in java options configuration section. The most commonly used setting is LENSES_HEAP_OPTS which restricts the memory usage of Lenses. The default value is -Xmx3g -Xms512m which permits Lenses to use as much as 3GB of memory for heap space.

Configuration Quirks¶

Lenses configuration is in HOCON format which at times can be challenging to convert to from other formats, such as configuration via environment variables, especially when these are set via nonstandard channels, such as yaml and docker environment files.

Yaml is well supported and multiline variables are supported. Quotes should be avoided unless they are needed as literals, like in the url and jmx sections of LENSES_ZOOKEEPER_HOSTS. An example follows:

environment:
  LENSES_PORT: 9991
  LENSES_KAFKA_BROKERS: PLAINTEXT://kafka-1:9092,PLAINTEXT://kafka-2:9092,PLAINTEXT://kafka-3:9092
  LENSES_ZOOKEEPER_HOSTS: |
      [
        {url:"zookeeper-1:2181",jmx:"zookeeper-1:9585"},
        {url:"zookeeper-2:2181",jmx:"zookeeper-2:9585"},
        {url:"zookeeper-3:2181",jmx:"zookeeper-3:9585"}
      ]
  LENSES_SCHEMA_REGISTRY_URLS: |
    [
      {url:"http://registry-1:8081",jmx:"registry-1:9582"},
      {url:"http://registry-2:8081",jmx:"registry-2:9582"}
    ]
  LENSES_CONNECT_CLUSTERS: |
    [
      {name:"production",
      urls: [
              {url:"http://connect-1:8083",jmx:"connect-1:9584"},
              {url:"http://connect-2:8083",jmx:"connect-2:9584"}
              {url:"http://connect-3:8083",jmx:"connect-3:9584"}
            ],
      statuses:"connect-statuses",
      configs:"connect-configs",
      offsets:"connect-offsets"}
    ]
  LENSES_ALERT_PLUGINS_SLACK_ENABLED: "false"
  LENSES_SECURITY_MODE: BASIC
  LENSES_SECURITY_GROUPS: |
      [
        {"name": "adminGroup", "roles": ["admin", "write", "read"]},
        {"name": "writeGroup", "roles": ["read", "write"]},
        {"name": "readGroup",  "roles": ["read"]},
        {"name": "nodataGroup",  "roles": ["nodata"]}
      ]
  LENSES_SECURITY_USERS: |
      [
        {"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
        {"username": "write", "password": "write", "displayname": "Write User", "groups": ["writeGroup"]},
        {"username": "read", "password": "read", "displayname": "Read Only", "groups": ["readGroup"]},
        {"username": "nodata", "password": "nodata", "displayname": "No Data", "groups": ["nodataGroup"]}
      ]

Docker environment files do not support multiline entries. Again, quotes should be used only when literals are expected and avoided in any other case.

LENSES_PORT=9991
LENSES_KAFKA_BROKERS=PLAINTEXT://kafka-1:9092,PLAINTEXT://kafka-3:9092,PLAINTEXT://kafka-3:9092
LENSES_ZOOKEEPER_HOSTS=[{url:"zookeeper-1:2181",jmx:"zookeeper-1:9585"},{url:"zookeeper-2:2181",jmx:"zookeeper-2:9585"},{url:"zookeeper-3:2181",jmx:"zookeeper-3:9585"}]
LENSES_SCHEMA_REGISTRY_URLS=[{url:"http://registry-1:8081",jmx:"registry-1:9582"},{url:"http://registry-2:8081",jmx:"registry-2:9582"}]
SD_CONFIG=provider=gce zone_pattern=europe-west1.*
SD_CONNECT_NAMES=production
SD_CONNECT_JMX_PORTS=9584
SD_CONNECT_FILTERS=tag_value=connect-worker
LENSES_SECURITY_GROUPS=[{"name": "adminGroup", "roles": ["admin", "write", "read"]}, {"name": "writeGroup", "roles": ["read", "write"]}]
LENSES_SECURITY_USERS=[{"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]}, {"username": "write", "password": "write", "displayname": "Write User", "groups": ["writeGroup"]}]

Monitoring and Prometheus¶

Lenses use the JVM, as such they can expose a JMX endpoint where applications can connect to access metrics. To enable the JMX endpoint, please set the environment variable LENSES_JMX_PORT. Depending on your enviroment, additional settings may be needed, such as:

LENSES_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=[LENSES_JMX_HOST] -Dcom.sun.management.jmxremote.rmi.port=[LENSES_JMX_PORT]"

A Prometheus endpoint is provided by default, through a jmx_exporter instance that is loaded as an agent into Lenses. Its port is 9102 and cannot be altered —but may be exposed under a different port as per docker settings. The Java agent is always loaded. It exposes the process and Kafka client metrics.

It is a common practice when deploying into Kubernetes to expose a liveness endpoint. Lenses docker image does not have a dedicated endpoint but the address of Lenses itself can be used for this purpose.

Plugins¶

Lenses can be extended via user-provided classes, such as custom de/serialization classes for records, user defined functions for the Lenses SQL table engine, LDAP filters and more. Learn more about plugins here.

In the docker, the LENSES_PLUGINS_CLASSPATH_OPTS is set automatically to /data/plugins. This path is exposed as a docker volume. You can add your jars there. Since Lenses scans frequently for new jars, you can add files to the volume even as the application runs, and they will be detected automatically. A simplistic example:

docker run -v /path/to/plugin/jar/directory:/data/plugins ...

It is still possible to set manually the LENSES_PLUGINS_CLASSPATH_OPTS (your setting takes precedence) or even add jars to the Lenses installation directory (/opt/lenses/plugins). However, the exposed docker volume (/data/plugins) is usually the best solution.

Please note that prior to version 2.2, only custom serde classes and LDAP filters were supported. The former was expected to be under /opt/lenses/serde or a user-defined path set in LENSES_SERDE_CLASSPATH_OPTS. These are still supported but will be deprecated in the future. If you use them, please switch to the new paths.

Cloud Service Discovery¶

In version 2.0 service discovery came into the Lenses docker as a preview feature.

Find out about cloud service discovery here.

Examples¶

These examples serve more as a quick reference guide.

Docker-compose example configuration.

version: '2'
services:
  lenses:
    image: landoop/lenses:2.2
    environment:
      LENSES_PORT: 9991
      LENSES_KAFKA_BROKERS: "PLAINTEXT://broker.1.url:9092,PLAINTEXT://broker.2.url:9092"
      LENSES_ZOOKEEPER_HOSTS: |
        [
          {url:"zookeeper.1.url:2181", jmx:"zookeeper.1.url:9585"},
          {url:"zookeeper.2.url:2181", jmx:"zookeeper.2.url:9585"}
        ]
      LENSES_SCHEMA_REGISTRY_URLS: |
        [
          {url:"http://schema.registry.1.url:8081",jmx:"schema.registry.1.url:9582"},
          {url:"http://schema.registry.2.url:8081",jmx:"schema.registry.2.url:9582"}
        ]
      LENSES_CONNECT_CLUSTERS: |
        [
          {
            name:"data_science",
            urls: [
              {url:"http://connect.worker.1.url:8083",jmx:"connect.worker.1.url:9584"},
              {url:"http://connect.worker.2.url:8083",jmx:"connect.worker.2.url:9584"}
            ],
            statuses:"connect-statuses-cluster-a",
            configs:"connect-configs-cluster-a",
            offsets:"connect-offsets-cluster-a"
          }
        ]
      LENSES_SECURITY_MODE: BASIC
      # Secrets can also be passed as files. Check _examples/
      LENSES_SECURITY_GROUPS: |
        [
          {"name": "adminGroup", "roles": ["admin", "write", "read"]},
          {"name": "readGroup",  "roles": ["read"]}
        ]
      LENSES_SECURITY_USERS: |
        [
          {"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]},
          {"username": "read", "password": "read", "displayname": "Read Only", "groups": ["readGroup"]}
        ]
    ports:
      - 9991:9991
      - 9102:9102
    volumes:
      - ./license.json:/license.json
    network_mode: host

A Kubernetes pod and service example is available at here. More information about running Lenses inside Kubernetes is available at the kubernetes and helm section.