Kubernetes

Lenses can be deployed on Kubernetes by leveraging its prebuilt docker image.

Helm, a package manager for Kubernetes, is the recommended way to install Lenses. Helm can be download from here and relies on kubectl. Helm and kubectl are not part of the Lenses package and must be installed separately.

For a current list of our existing Helm Charts please visit our repo and are available on our GitHub repo.

Helm Chart

The Helm chart will optionally configure:

RBAC
Kubernetes role-based access controls. The chart will install the required cluster roles and cluster role bindings.
Services
The chart will create a service to abstract access to the Lenses backend.
Ingress
Lenses exposes a user interface and the endpoints behind. To allow access from outside Kubernetes, the chart can optionally create an ingress. The ingress is expected to be based on an Ingress Controller such as Traefik. Right now, we don’t support path-based routing.
Secrets
Sensitive data such as usernames, passwords, and TLS information should be distributed to pods by Kubernetes secrets. The chart will create secrets for the pod to mount. Secrets are used for the JKS stores to avoid base64 encoding twice the binaries via a config map.
ConfigMaps
A config map is used to hold krb5.conf data for SASL.
Deployment
The chart will deploy Lenses as a Deployment resource, mounting secrets in /mnt/secrets. The pods will be annotated with Prometheus annotations to allow automatic collection of the JMX metrics Lenses exposes.
Labels
All resources deployed by the chart are labeled with lenses.io./app: <release name> and lenses.io/app.type: lenses-<type>
Annotations
The Lenses deployment resource has the Prometheus annotations attached. Metrics are exposed on port 9102 at /metrics

Installing via Helm

Add the Lenses Helm chart repo:

# Add repos other connector charts
helm repo add lensesio https://lensesio.github.io/kafka-helm-charts/

Use helm to install Lenses via the command line:

helm install lensesio/lenses --name lenses --namespace lenses

Example to override the defaults i.e. enabled ingress and SSL:

helm install lensesio/lenses --name lenses --namespace lenses --set ingress.enabled=true

Values Configuration

Field Description Type Default Required
image.repository The Lenses image landoop/lenses    
image.tag The Lenses image tag 2.3    
replicaCount Number of pods/instances to deploy int 1 yes
monitoring.enabled
Flag to add Prometheus
annotations to the pod for metric
scraping
boolean true no
monitoring.pipeline
A label to attach to
the pods for include in JMX
string   yes
monitoring.port
The port the Prometheus
JMX exporter is exposing metrics on
int 9102 no
monitoring.path
The path which the metrics
are served on
string /metrics no
resources.limit.memory
The kubernetes resource
memory limit allowed for the pod
string 5Gi no
resources.request.memory
The kubernetes resource
memory request allowed for the pod
string 4Gi no
rbacEnable
Create cluster roles and
bindings if the kubernetes cluster
has RBAC enabled
boolean true  
restPort
The port for Lenses
int 3030  
serviceAccount
The Kubernetes service account
Lenses will use to deploy
resources. It must be
patched with the image pull
secret for the Lenses SQL runners
string default no
persistence.enabled Create a persistence volume claim boolean false no
persistence.accessModes
The access modes which the
persistence volume can be accessed
list   yes
persistence.existingClaim
if it’s true will use your
own Persistent volume claim
boolean false yes
persistence.storageClass
If set to ‘-‘, storageClassName: ‘’,
it will disable the dynamic provisioning
If undefined (the default) or set to null,
no storageClassName spec is set
choosing the default provisioner.
(gp2 on AWS, standard on GKE, Azure, Openstack)
string ‘-‘ no
persistence.size The size of the persistence volume string 5Gi yes
service.enabled
Create a service for Lenses
boolean true no
service.type Kubernetes service type string ClusterIP no
service.annotations
Annotations to
add to the service
map   no
ingress.enabled
Create an ingress resource
boolean true no
ingress.host
The host or ip
of the external load balancer
(Minikube ip)
string   yes
ingress.annotations
The annotations to
add to the ingress resource
map
kubernetes.io/ingress.class: traefik,
traefik.frontend.rule.type: PathPrefixStrip
no
ingress.tls.enabled
Enable TLS ingress
boolean false no
ingress.tls.crt
The TLS certificate
data, base64 encoded
Base64 string   no
ingress.tls.key
The TLS key data
base64 encoded
Base 64 string   no
lenses.lensesOpts
For additional generic JVM settings
string   no
lenses.jvm.heapOpts
JVM heap options for Lenses
string   no
lenses.jvm.performanceOpts
JVM performance tuning
extra options for Lenses
string   no
lenses.jvm.logBackOpts
Logback options
string   no
lenses.jvm.trustStoreFileData
The base64 encoded contents of the truststore
base64   no
lenses.jvm.trustStorePassword
The truststore password
string   no
lenses.kafka
List of Kafka brokers.
string   yes
lenses.zookeepers.enabled Flag to enable Zookeeper boolean   no
lenses.zookeepers
List of Zookeepers.
string   yes
lenses.schemaRegistries
List of Schema Registries.
string   yes
lenses.connectClusters
Array of Connect Clusters.
string   yes
lenses.grafanaUrl The url to the grafana host string   no
lenses.topics.suffix
A suffix to add to Lenses
system topics
string   no
lenses.sql string   no
lenses.security.ldap.enabled Flag to enable LDAP string   no
lenses.security.ldap.url   string   no
lenses.security.ldap.base   string   no
lenses.security.ldap.password LDAP password for the user string   no
lenses.security.ldap.user LDAP user string   no
lenses.security.ldap.plugin.class
FQDN class for the ldap plugin
to use
string   no
lenses.security.ldap.plugin.memberOfKey   string   no
lenses.security.ldap.plugin.groupExtractRegex   string   no
lenses.security.ldap.plugin.personNameKey   string   no
lenses.security.kerberos.enabled Flag to enable kerberos string   no
lenses.security.kerberos.service.principal The kerberos principal string   no
lenses.security.kerberos.keyTabData The kerberos keytab data string   no
lenses.security.kerberos.debug Enable kerberos debug boolean   no
lenses.license The Lenses license data string   no
lenses.licenseUrl Url to fetch the license from string   no
lenses.configOverrides
Extra configs. Allows
for any extra configurations
For example
lenses.interval.summary=1000
list of dictionaries   no

Bootstrap servers

lenses.brokers contains the broker options. Multiple brokers are supported.

If you are deploying brokers inside Kubernetes they should be deployed as a statefulset like this. This allows the pods to have stable network identifiers. Each pods address should be added as an entry. The address takes the form of:

<statefulset-name>-<pod ordinal identifier>.<service name>.<namespace>.svc.cluster.local

For example, a statefulset of 3 replicas called broker with a headless service called broker, the addresses would be:

broker-0.broker.defaut.svc.cluster.local
broker-1.broker.defaut.svc.cluster.local

Note

New brokers added or removed on scaling will not be reflected. Currently, Lenses will require a config update. Future releases will address this.

If you only have one broker you can set the service name.

If your brokers are outside Kubernetes add hostnames.

Field Description Type Default Required
ssl.enabled SSL is enabled on the brokers false    
ssl.truststoreFileData
The base64 encoded contents
of the truststore
base64   no
ssl.keystoreFileData
The base64 encoded contents
of the keystore
base64   no
ssl.truststorePassword The truststore password string   no
ssl.keystorePassword The keystore password string   no
sasl.enabled SASL is enabled on the brokers boolean false no
sasl.keyTabData
The base64 encoded contents
of the keytab file is sasl enabled with GSSAPI
base64   no
sasl.jaasFileData
The contents of the jaas.conf file
is sasl is enabled
string   no
sasl.mechanism
The security.mechanism to use.
GSSAPI, SCRAM or PLAINTEXT
string GSSAPI no
sasl.krb5Conf
The contents of the krb5Conf file

if the sasl mechanism is GSSAPI

string   no

bootstrapServers is a list of brokers, names and ports

Field Description Type Default Required
bootstrapServers.name Host name of the broker      
bootstrapServers.port The PLAINTEXT default Kafka port int 9092 yes
bootstrapServers.sslPort The SSL Kafka port int 9093 yes
bootstrapServers.saslPort The SASL_SSL Kafka port int 9094 yes
bootstrapServers.saslPlainTextPort The SASL_PLAINTEXT Kafka port int 9095 yes

metrics used to monitor the health of your cluster and show metrics and information

Field Description Type Default Required
metrics.type The type of metrics which will be used eg. JMX, JOLOKIAP, JOLOKIAG string   no
metrics.ssl Enable SSL if metryics type has enabled boolean false no
metrics.username The username when metrics are secured string   no
metrics.password The password when metrics are secured string   no
metrics.ports List of port, host and broker ID to monitor list   no

Example:

lenses:
  kafka:
    ssl:
      enabled: false
      trustStoreFileData:
      keyStoreFileData:
      trustStorePassword:
      keyStorePassword:
      keyPassword:
    sasl:
      enabled: false
      # keyTabData is the base64 encoded contents kerberos keytab file is using kerberos mounted in /mnt/secrets
      keyTabData: |-

      # jaasFileData is the contents of the kafka jaas file mounted in /mnt/secrets
      jaasFileData: |-

      # mechanism is the sasl authentication mechanism GSSAPI, SCRAM or PLAIN
      mechanism: "GSSAPI"
      # krb5Conf is the Kerberos config data to be mounted into /etc
      krb5Conf: |-

    metrics:
      type: "JMX"
      ssl: true
      username: admin
      password: admin
      port:
        - id: 1
          port: 9581
          host: "host1"
        - id: 2
          port: 9581
          host: "host2"

    bootstrapServers:
      - name: kafka
        port: 9092
        sslPort: 9093
        saslSslPort: 9094
        saslPlainTextPort: 9095

Zookeepers

lenses.zookeepers is a list of zookeepers detailing the hostname, HTTP protocol, and ports. Multiple zookeepers are supported.

If you are deploying a quorum of zookeepers inside Kubernetes they should be deployed as a statefulset like this. This allows the pods to have stable network identifiers. Each pods address should be added as an entry. The address takes the form of:

<statefulset-name>-<pod ordinal identifier>.<service name>.<namespace>.svc.cluster.local

For example, a statefulset of 3 replicas called zookeeper with a headless service called zookeeper, the addresses would be:

zookeeper-0.zookeeper.defaut.svc.cluster.local
zookeeper-1.zookeeper.defaut.svc.cluster.local

Note

New zookeepers added or removed on scaling will not be reflected. Currently, Lenses will require a config update. Future releases will address this.

If you only have one zookeeper you can set the service name.

If your zookeepers are outside Kubernetes add hostnames.

Field Description Type Default Required
host The hostname of the zookeeper instance string zookeeper yes
port The port for the zookeeper instance int 8081 yes
metrics.type The type of metrics which will be used eg. JMX, JOLOKIAP, JOLOKIAG string   no
metrics.ssl Enable SSL if metrics type has enabled boolean false no
metrics.username The username when metrics are secured string   no
metrics.password The password when metrics are secured string   no
metrics.port The metrics port for zookeeper list   no

Example:

lenses:
  zookeepers:
    hosts:
      - host: zookeeper-1
        port: 2181
        metrics:
          type: "JMX"
          port: 9102
          username:
          password:
          ssl:
      - host: zookeeper-2
        port: 2181
        metrics:
          type: "JMX"
          port: 9102
          username: admin
          password: admin
          ssl: true

Schema Registries

lenses.schemaRegistries.hosts is a list of schemaRegistries detailing the hostname, HTTP protocol and ports. Multiple schema registries are supported.

If you are deploying multiple schema registries for high availability inside Kubernetes they should be deployed as a statefulset. This allows the pods to have stable network identifiers. Each pods address should be added as an entry. The address takes the form of:

<statefulset-name>-<pod ordinal identifier>.<service name>.<namespace>.svc.cluster.local

For example, a statefulset of 2 replicas called schema-registry with a headless service called schema-registry, the addresses would be:

schema-registry-0.schema.default.svc.cluster.local
schema-registry-1.schema.default.svc.cluster.local

Note

New schema registries added or removed on scaling will not be reflected. Currently, Lenses will require a config update. Future releases will address this.

If you only have one schema registry you can set the service name.

If your schema registries are outside Kubernetes add hostnames.

Field Description Type Default Required
enabled Enable schema registry support boolean false  
host The host name of the schema registry instance string schema-registry yes
protocol The HTTP protocol, http or https string http yes
port The port for the schema registry instance int 2181 yes
metrics.type
The type of metrics which will be used
eg. JMX, JOLOKIAP, JOLOKIAG
string   no
metrics.ssl Enable SSL if metrics type has enabled boolean false no
metrics.username The username when metrics are secured string   no
metrics.password The password when metrics are secured string   no
metrics.port The metrics port for zookeeper list   no
security.enabled Enable schema registry security support boolean false  
security.authType
The Worker authentication type
URL, NONE, SASL_INHEIRT
or USER_INFO
string   no
security.username
Username to authentication
with the workers
string   no
security.password
Password to authentication
with the workers
string   no
security.kerberos.enabled Enable schema registry security Kerberos support boolean false  
security.kerberos.principal
The principal which will
be used for Kerberos SPNEGO eg. HTTP@myservice
string false  
security.kerberos.keyTabData
The base64 encoded contents
of the keytab file is sasl enabled with GSSAPI
base64   no
security.kerberos.jaasFileData
The contents of the jaas.conf file
is sasl is enabled
string   no

Example:

lenses:
  schemaRegistries:
    enabled: true
    hosts:
      - host: schema-registry-1
        protocol: http
        port: 8081
        metrics:
          type: "JMX"
          port: 9102
          username:
          password:
          ssl:
      - host: schema-registry-2
        protocol: http
        port: 8081
        metrics:
          type: "JMX"
          port: 9102
          username:
          password:
          ssl:

Connect clusters

The lenses.connectClusters is an array of Connect clusters detailing the cluster name, ports, backing topics and worker hosts. Multiple clusters can be configured.

If you are deploying a Connect cluster inside Kubernetes they should be deployed as a statefulset. This allows the pods to have stable network identifiers. Each pods address should be added as an entry. The address takes the form of:

<statefulset-name>-<pod ordinal identifier>.<service name>.<namespace>.svc.cluster.local

For example, a statefulset of 2 replicas called connect with a headless service called connect, the addresses would be:

connect-worker-0.connect.defaut.svc.cluster.local
connect-worker-1.connect.defaut.svc.cluster.local

Note

New workers added or removed on scaling will not be reflected. Currently, Lenses will require a config update. Future releases will address this.

If you only have one zookeeper you can set the service name.

If your connect workers are outside Kubernetes add hostnames.

Field Description Type Default Required
enabled Enabled Kafka Connect support boolean false yes
name The name/alais for the cluster string   yes
port The workers rest port int 8083 yes
protocol The HTTP protocol, http or https string http yes
auth
The Worker authentication type
URL, NONE, SASL_INHEIRT
or USER_INFO
string NONE no
username
Username to authentication
with the workers
string   no
password
Password to authentication
with the workers
string   no
offsetsTopic
The offset backing topic
for the cluster
string connect-offsets yes
statusTopic
The statuses backing topic
for the cluster
string connect-statuses yes
configsTopic
The configs backing topic
for the cluster
string connect-configs yes
hosts List of connect worker host names string   yes
metrics.type
The type of metrics which will be used
eg. JMX, JOLOKIAP, JOLOKIAG
string   no
metrics.ssl Enable SSL if metrics type has enabled boolean false no
metrics.username The username when metrics are secured string   no
metrics.password The password when metrics are secured string   no
metrics.port The metrics port for zookeeper list   no

Example:

connectClusters:
  enabled: true
  clusters:
    - name: datascience
      port: 8083
      jmxPort: 9102
      offsetsTopic: connect-offsets-datascience
      statusTopic: connect-statuses-datascience
      configTopic: connect-configs-datascience
      hosts:
        - host: worker-ds-1
          metrics:
            type: "JMX"
            port: 9102
            username:
            password:
            ssl:
        - host: worker-ds-1
          metrics:
            type: "JMX"
            port: 9102
            username:
            password:
            ssl:
        - host: worker-ds-1
          metrics:
            type: "JMX"
            port: 9102
            username:
            password:
            ssl:

    - name: dataengineering
      port: 8083
      jmxPort: 9102
      offsetsTopic: connect-offsets-dataengineering
      statusTopic: connect-statuses-dataengineering
      configTopic: connect-configs-dataengineering
      hosts:
        - worker-de-1
        - worker-de-1
        - worker-de-1
      auth: URL
      username: myusername
      password: mypassword

SQL Processor Options

lenses.sql contains the options for configuring SQL Processors deployed via Lenses.

Field Description Type Default Required
processorImage
The processor image to use
for Lenses SQL runners
string eu.gcr.io/k8-engine/lenses-sql-processor no
processorImageTag The processor image tag string 2.1.0 no
mode The SQL scaling mode, IN_PROCESS, CONNECT or KUBERNETES string IN_PROCESS yes
heap The heap space for the sql processors string 1024M no
memLimit
The Kubernetes memory resource
limit for the sql processors
string 1152M no
memRequest
The Kubernetes memory resource
request for the sql processors
string 128M no
truststoreFileData
The base64 encoded contents
of the truststore
base64   no
keystoreFileData
The base64 encoded contents
of the keystore
base64   no
truststorePassword The truststore password string   no
keystorePassword The keystore password string   no
enabled SASL is enabled on the brokers boolean false no
keyTabData
The base64 encoded contents
of the keytab file is sasl enabled with GSSAPI
base64   no
jaasFileData
The contents of the jaas.conf file
is sasl is enabled
string   no
sasl.mechanism
The security.mechanism to use.
GSSAPI, SCRAM or PLAINTEXT
string GSSAPI no
sasl.krb5Conf
The contents of the krb5Conf file

if the sasl mechanism is GSSAPI

string   no
Alert Plugins

lenses.alerts is a list of Alert Plugins. Multiple alert plugins are supported.

Field Description Type Default Required
enabled Enable alert plugins integration boolean false no
consumersLagThreshold
The default consumer
lag for alerting
int 50000  
plugins The host name of the alertmanagers instance list   no

Example:

alerts:
  enabled: true
  consumersLagThreshold: 50000

  plugins:
    - class: "io.lenses.alerts.plugin.slack.SlackAlertsPlugin"
      config: |
        {
          webhook-url="https://your-slack-webhook-url
          username=Lenses™
          channel=lenses-slack-test
        }
    - class: "io.lenses.alerts.plugin.am.AlertManagerPlugin"
      config: |
        {
          endpoints="http://your-alert-manager-url"
          source=PROD
          generator-url="http://lenses_prod:port1"
        }
Security Default user

The security default user section contains the default username and password for Lenses:

Field Description Type Default Required
lenses.security.user Default User name string admin no
lenses.security.password Default User password string admin no

example:

lenses:
  security:
    defaultUser:
      username: admin
      password: admin

Single Sign On (SSO - SAML)

Lenses supports Single Sign On (SSO) authentication using the SAML 2.0 protocol. SSO allows you to authenticate and login your users using your organisation’s Identity Provider (IdP), refer here.

Field Description Type Default Required
lenses.security.saml.enabled Flag to enable saml boolean   no
lenses.security.saml.baseUrl The base URL of Lenses string   no
lenses.security.saml.provider The SSO provider string   no
lenses.security.saml.keyStoreFileData
The base64 encoded contents
of the keystore
base64   no
lenses.security.saml.keyStorePassword The keystore password string   no
lenses.security.saml.keyPassword
The password to use for the SAML
generated key within the keystore
string   no
lenses.security.saml.metadataFileData
The base64 encoded contents
of the XML File provided by the external Identity Provider
base64   no

Persistence Volume

You can enable Persistence Volume (PV) to keep the state for the Data Policies module. The chart will create automatically a Persistence Volume Claim (PVC) and if you provide a Storage Class, it will create/bound dynamically a PV with a reclaim policy Delete. Based on the type of storageClass you will use, the PVC will choose the appropriate volume plugin to be used for provisioning PVs. You can check more details here.

If you need to change the re-claim policy from Delete to a different one eg. Retain you can use the following command:

kubectl patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

LoadBalancer Service

You can also set the service.serviceType to LoadBalancer and set a static address. To do this you must comment out the LoadBalancer section and set the values accordingly.

Secrets

Kafka Secrets

kafka.ssl and kafka.sasl values are placed in a Secret called <release-name>. Both the key/trust stores and the passwords are added. The items from this secret are then mounted in /mnt/secrets.

Take care with SASL enabled Kafka clusters to set the path correctly in the JAAS config data to /mnt/secrets for the keytab file. The kafka.sasl.keyTabData is mounted as a /mnt/secrets/keytab. If sasl is enabled -Djava.security.auth.login.config=/mnt/secrets/jaas.conf is added to the Lenses java opts.

Lenses secrets are labelled lenses.io/app.type: lenses-secret.

Lenses Secrets
Lenses secrets, lenses.license and lenses.security** values are placed in a secret named after the helm release.

ConfigMaps

The chart will create a configMap to hold the krb5.conf for Lenses and the SQL processors. This will be mounted in /etc.

Lenses configMaps are labelled lenses.io/app.type: lenses-configmap.

RBAC

If your Kubernetes cluster has RBAC enabled then Lenses service account Lenses requires the following access:

Resource Verbs
namespaces list, watch, get, create
pods list, watch, get, update, create, patch, delete
deployments list, watch, get, update, create, patch, delete
replicaSet list, watch, get, update, create, patch, delete
events list, watch, get

The Lenses Helm chart will create the required cluster roles and bindings.

Lenses cluster role bindings are labeled lenses.io/app.type: lenses-cluster-role-binding and the cluster role lenses.io/app.type: lenses-cluster-role.

More Kubernetes integration

Lenses can deploy SQL processors on Kubernetes. Please see here.