4.0

You are viewing documentation for an older version of Lenses.io View latest documentation here

Helm

The official Helm chart is available on GitHub . Find bellow documentation and examples:

KeyDescriptionTypeDefaultRequired
replicaCountNumber of pods/instances to deployint1yes
image.repositoryThe Lenses docker image namestringlensesio/lensesno
image.tagThe Lenses docker image tagstring4.0no
monitoring.enabledSet to true to add add Prometheus annotations to the pod for metric scrapingbooleantrueno
monitoring.pipelineLabel to attach to the pods for JMX monitoringstringlensesno
monitoring.portThe port for Prometheus JMX metrics exporterint9102no
monitoring.pathThe path that exposes the prometheus metricsstring/metricsno
resources.limits.memoryThe k8 resource memory limit for the podstring5Gino
resources.requests.memoryThe k8 resource memory request for the podstring4Gino
rbacEnableIf k8 cluster has RBAC enabled, create cluster roles and bindingsbooleantrueno
restPortThe port for Lenses APIs and UIint3030no
servicePortThe service portint80no
serviceAccountThe k8 service account Lenses will use to deploy resources. It must be patched with the image pull secret for Streaming SQL on K8stringdefaultno
persistence.enabledCreate a persistence volume claimbooleanfalseno
persistence.accessModesThe access modes for the persistence volumelistReadWriteOnceyes
persistence.sizeThe size of the persistence volumestring5Giyes
persistence.existingClaimIf true use your own Persistent volume claimbooleanfalseyes
persistence.storageClassIf set to ‘-’, storageClassName: "" it disables dynamic provisioning. If undefined or set to null, no storageClassName spec is set and chooses the default provisioner (gp2 on AWS, standard on GKE, Azure, Openstack)stringno
service.enabledCreate a service for Lensesbooleantrueno
service.typeK8 service typestringClusterIPno
service.annotationsAnnotations to add to the servicemap{}no
ingress.enabledIf true an ingress resource will be createdbooleanfalseno
ingress.hostThe host or ip of the external load balancerstringyes
ingress.annotationsThe annotations to add to the ingress resourcemapkubernetes.io/ingress.class: traefik
ingress.tls.enabledEnable ingress TLSbooleanfalseno
ingress.tls.crtThe ingress TLS certificate base64 encodedbase64
ingress.tls.keyThe ingress TLS key base64 encodedbase64no
lenses.lensesOptsFor additional generic JVM settingsstringno
lenses.jvm.heapOptsAdditional JVM heap options for Lensesstringno
lenses.jvm.logBackOptsAdditional Logback optionsstringno
lenses.jvm.performanceOptsAdditional JVM performance options for Lensesstringno
lenses.opts.trustStoreFileDataThe contents of the truststore base64 encodedbase64no
lenses.opts.trustStorePasswordThe truststore password base64 encodedbase64no
lenses.opts.keyStoreFileDataThe contents of the keystore base64 encodedbase64no
lenses.opts.keyStorePasswordThe keystore password base64 encodedbase64no
lenses.append.confAdditional custom values will be appended to lenses configurationstringno
lenses.kafkaList of Kafka brokersstringyes
lenses.zookeepers.enabledEnable Zookeeper connection (optional)booleanno
lenses.zookeepersList of Zookeepersstringyes
lenses.schemaRegistriesList of Schema Registriesstringyes
lenses.connectClustersList of Connect Clustersstringyes
lenses.sqlSee Streaming SQL optionsstringno
lenses.grafanaUrlThe url of Grafanastringno
lenses.topics.suffixSuffix to add to system topicsstringno
lenses.security.ldap.enabledEnable LDAP. Learn more for LDAPbooleanfalseno
lenses.security.ldap.urlThe LDAP urlstringno
lenses.security.ldap.baseThe LDAP user search base settingsstringno
lenses.security.ldap.filterThe LDAP user search filterstringno
lenses.security.ldap.userThe LDAP usernamestringno
lenses.security.ldap.passwordThe LDAP password for the above userstringno
lenses.security.ldap.plugin.classFQDN class for the LDAP pluginstringno
lenses.security.ldap.plugin.memberOfKeystringno
lenses.security.ldap.plugin.groupExtractRegexstringno
lenses.security.ldap.plugin.personNameKeystringno
lenses.security.kerberos.enabledFlag to enable Kerberosstringno
lenses.security.kerberos.service.principalThe Kerberos principalstringno
lenses.security.kerberos.keyTabDataThe Kerberos keytab datastringno
lenses.security.kerberos.debugEnable Kerberos debugbooleanno
lenses.security.append.confAdditional custom values will be appended to lenses security configurationstringno
lenses.sidecarContainersList of sidecar containers ( API )listno
lenses.licenseThe Lenses license datastringno
lenses.licenseUrlUrl to fetch the license fromstringno
lenses.configOverridesAllows for additional configurations i.e. lenses.interval.summary=1000list of dictionariesno

License 

Add the contents of the license file under the lenses.license YAML property:

# Add the contents of your license file
lenses:
    license: |-
                {"source":"Lenses.io","clientId":"123ab789-abc..","key":"eyJhb.."}

Bootstrap servers 

lenses.brokers contains the broker options. Multiple brokers are supported.

KeyDescriptionTypeDefaultRequired
ssl.enabledSSL is enabled on the brokersbooleanfalseno
ssl.truststoreFileDataThe base64 encoded contents of the truststorebase64no
ssl.keystoreFileDataThe base64 encoded contents of the keystorebase64no
ssl.truststorePasswordThe truststore passwordstringno
ssl.keystorePasswordThe keystore passwordstringno
sasl.enabledIf SASL is enabled on the brokersbooleanfalseno
sasl.keyTabDataThe base64 encoded contents of the keytab file if sasl enabled with GSSAPIbase64no
sasl.jaasFileDataThe contents of the jaas.conf file if SASL is enabledstringno
sasl.mechanismThe security.mechanism to use.
GSSAPI, SCRAM or PLAINTEXT
stringGSSAPIno
sasl.krb5ConfThe contents of the krb5Conf file if the SASL mechanism is GSSAPIstringno

bootstrapServers is a list of brokers, names and ports.

KeyDescriptionTypeDefaultRequired
bootstrapServers.nameHost name of the brokerstringyes
bootstrapServers.portThe PLAINTEXT default Kafka portint9092yes
bootstrapServers.sslPortThe SSL Kafka portint9093yes
bootstrapServers.saslPortThe SASL_SSL Kafka portint9094yes
bootstrapServers.saslPlainTextPortThe SASL_PLAINTEXT Kafka portint9095yes

metrics used to monitor the health of your cluster and show metrics and information.

FieldDescriptionTypeDefaultRequired
metrics.typeThe type of metrics to use: JMX, AWS, JOLOKIAP or JOLOKIAGstringno
metrics.sslEnable SSL if metrics type has enabledbooleanfalseno
metrics.usernameThe username when metrics are securedstringno
metrics.passwordThe password when metrics are securedstringno
metrics.portDefault port to monitor for all brokersintno
metrics.portsList of port, host and broker ID to override default for particular brokerlistno

Example:

lenses:
  kafka:
    ssl:
      enabled: false
      trustStoreFileData:
      keyStoreFileData:
      trustStorePassword:
      keyStorePassword:
      keyPassword:
    sasl:
      enabled: false
      # keyTabData is the base64 encoded contents kerberos keytab file is using kerberos mounted in /mnt/secrets
      keyTabData: |-

      # jaasFileData is the contents of the kafka jaas file mounted in /mnt/secrets
      jaasFileData: |-

      # mechanism is the sasl authentication mechanism GSSAPI, SCRAM or PLAIN
      mechanism: "GSSAPI"
      # krb5Conf is the Kerberos config data to be mounted into /etc
      krb5Conf: |-

    metrics:
      type: "JMX"
      ssl: true
      username: admin
      password: admin
      ports:
        - id: 1
            port: 9581
            host: "host1"
        - id: 2
            port: 9581
            host: "host2"

    bootstrapServers:
      - name: kafka
      port: 9092
      sslPort: 9093
      saslSslPort: 9094
      saslPlainTextPort: 9095

Zookeepers 

If you have one of more Zookeeper services, use the lenses.zookeepers.hosts options.

KeyDescriptionTypeDefaultRequired
hostThe hostname of the Zookeeper instancestringzookeeperyes
portThe port for the zookeeper instanceint2181yes
metrics.typeThe type of metrics to use: JMX, JOLOKIAP or JOLOKIAGstringno
metrics.sslEnable SSL if metrics type has enabledbooleanfalseno
metrics.usernameThe username when metrics are securedstringno
metrics.passwordThe password when metrics are securedstringno
metrics.portThe metrics port for zookeeperintno

Example:

lenses:
  zookeepers:
    hosts:
      - host: zookeeper-1
        port: 2181
        metrics:
          type: "JMX"
          port: 9102
          username:
          password:
          ssl:
      - host: zookeeper-2
        port: 2181
        metrics:
          type: "JMX"
          port: 9102
          username: admin
          password: admin
          ssl: true

Schema Registries 

If you have one of more Schema Registry services, use the lenses.schemaRegistries options.

KeyDescriptionTypeDefaultRequired
enabledEnable schema registry supportbooleanfalseno
hostThe host name of the schema registry instancestringschema-registryyes
protocolThe HTTP protocol, http or httpsstringhttpyes
portThe port of the schema registry instanceint2181yes
metrics.typeThe type of metrics to use: JMX, JOLOKIAP or JOLOKIAGstringno
metrics.sslUse SSL to fetch metricsbooleanfalseno
metrics.usernameThe username when metrics are securedstringno
metrics.passwordThe password when metrics are securedstringno
metrics.portThe metrics port for Schema Registryintno
security.enabledEnable schema registry security supportbooleanfalseno
security.authTypeAuthentication type: NONE, URL, USER_INFO or SASL_INHERITstringno
security.usernameAuthentication usernamestringno
security.passwordAuthentication passwordstringno
security.kerberos.enabledEnable schema registry Kerberos supportbooleanfalseno
security.kerberos.principalThe principal to use for Kerberos SPNEGO eg. HTTP@myservicestringfalseno
security.kerberos.keyTabDataThe base64 encoded contents of the keytab file if SASL enabled with GSSAPIbase64no
security.kerberos.jaasFileDataThe contents of the jaas.conf file if SASL is enabledstringno

Example:

lenses:
  schemaRegistries:
    enabled: true
    hosts:
      - host: schema-registry-1
        protocol: http
        port: 8081
        metrics:
          type: "JMX"
          port: 9102
          username:
          password:
          ssl:
      - host: schema-registry-2
        protocol: http
        port: 8081
        metrics:
          type: "JMX"
          port: 9102
          username:
          password:
          ssl:

Kafka Connect clusters 

If you have one of more Kafka Connect clusters, us the lenses.connectClusters options and define your Connect clusters providing the cluster name, ports, backing topics and worker hosts.

KeyDescriptionTypeDefaultRequired
enabledEnable Kafka Connect supportbooleantrueyes
nameThe name/alias for the clusterstringyes
hostsList of connect worker host nameslistyes
portThe connect worker rest portint8083yes
protocolThe HTTP protocol, http or httpsstringhttpyes
offsetsTopicThe offset backing topic for the clusterstringconnect-offsetsyes
statusTopicThe statuses backing topic for the clusterstringconnect-statusesyes
configsTopicThe configs backing topic for the clusterstringconnect-configsyes
authTypeThe Worker authentication type
URL, NONE, SASL_INHERIT or USER_INFO
stringNONEno
usernameUsername to authenticate with the workersstringno
passwordPassword to authenticate with the workersstringno
metrics.typeThe type of metrics to use: JMX, JOLOKIAP or JOLOKIAGstringno
metrics.sslGet metrics via SSLbooleanfalseno
metrics.usernameThe username when metrics are securedstringno
metrics.passwordThe password when metrics are securedstringno
metrics.portThe metrics port for the workerslistno

Example:

connectClusters:
  enabled: true
  clusters:
    - name: datascience
      protocol: http
      port: 8083
      jmxPort: 9102
      offsetsTopic: connect-offsets-datascience
      statusTopic: connect-statuses-datascience
      configTopic: connect-configs-datascience
      hosts:
        - host: worker-ds-1
          metrics:
            type: "JMX"
            port: 9102
            username:
            password:
            ssl:
        - host: worker-ds-1
          metrics:
            type: "JMX"
            port: 9102
            username:
            password:
            ssl:
        - host: worker-ds-1
          metrics:
            type: "JMX"
            port: 9102
            username:
            password:
            ssl:

    - name: dataengineering
      protocol: http
      port: 8083
      jmxPort: 9102
      offsetsTopic: connect-offsets-dataengineering
      statusTopic: connect-statuses-dataengineering
      configTopic: connect-configs-dataengineering
      hosts:
        - host: worker-de-1
        - host: worker-de-1
        - host: worker-de-1
      auth: URL
      username: myusername
      password: mypassword

Streaming SQL options 

To configure Streaming SQL options for Lenses use the lenses.sql options:

KeyDescriptionTypeDefaultRequired
modeThe Streaming SQL scaling mode, IN_PROCESS, CONNECT or KUBERNETESstringIN_PROCESSyes
processorImageThe image to use for Streaming SQLstringlensesioextra/sql-processorno
processorImageTagThe image tagstring4.0no
heapThe heap space for the sql processorsstring900Mno
memLimitThe Kubernetes memory resource limit for the sql processorsstring1152Mno
memRequestThe Kubernetes memory resource request for the sql processorsstring128Mno
truststoreFileDataThe base64 encoded contents of the truststorebase64no
keystoreFileDataThe base64 encoded contents of the keystorebase64no
truststorePasswordThe truststore passwordstringno
keystorePasswordThe keystore passwordstringno
enabledIf SASL is enabled on the brokersbooleanfalseno
keyTabDataThe base64 encoded contents of the keytab file if sasl enabled with GSSAPIbase64no
jaasFileDataThe contents of the jaas.conf file if sasl is enabledstringno
sasl.mechanismThe security.mechanism to use.
GSSAPI, SCRAM or PLAINTEXT
stringGSSAPIno
sasl.krb5ConfThe contents of the krb5Conf file if the sasl mechanism is GSSAPIstringno

Example:

lenses:
  sql:
    mode: KUBERNETES
    heap: 900M
    memLimit: 1152M
    memRequest: 128M
    ssl:
      trustStoreFileData: |-

      keyStoreFileData: |-

      trustStorePassword:
      keyStorePassword:
      keyPassword:

Lenses HTTPS (TLS) 

lenses.tls contains tls options for configuring Lenses with TLS termination.

KeyDescriptionTypeDefaultRequired
lenses.tls.enabledEnable TLS Terminationbooleanfalseno
lenses.tls.truststoreFileDataThe base64 encoded contents of the truststorebase64no
lenses.tls.keystoreFileDataThe base64 encoded contents of the keystorebase64no
lenses.tls.truststorePasswordThe base64 encoded truststore passwordbase64no
lenses.tls.keystorePasswordThe base64 encoded keystore passwordbase64no
lenses.tls.keyPasswordThe password for the key within the keystorebase64no
lenses.tls.clientAuthEnable client auth via tlsbooleanfalseno

Example:

lenses:
  tls:
    enabled: true
    # base64 encoded keystore data
    # openssl base64 < keystore.jks | tr -d '\n'
    keyStoreFileData: |-
            /u3+7QAAAAIAAAACAAAAAgAGY2Fyb290...==
    # base64 keystore password
    # echo "$password" | tr -d '\n' | base64
    keyStorePassword: |-
            YWRtaW4xMjM0
    # base64 encoded truststore data
    # openssl base64 < truststore.jks | tr -d '\n'
    trustStoreFileData: |-
            /u3+7QAAAAIAAAABAAAAAgAGY2Fyb290...==
    # base64 truststore password
    # echo "$password" | tr -d '\n' | base64
    trustStorePassword: |-
            YWRtaW4xMjM0
    # base64 key password
    # echo "$password" | tr -d '\n' | base64
    keyPassword: |-
            YWRtaW4xMjM0
    clientAuth: false

Single Sign On ( SAML 2.0 ) 

Users can authenticate via SSO (Single Sign On) using the SAML 2.0 protocol using one of the supported integrations . To use SSO remember to also enable TLS .

KeyDescriptionType
lenses.security.saml.enabledSet true to enable SAML 2.0boolean
lenses.security.saml.baseUrlThe base URL of Lensesstring
lenses.security.saml.providerThe SAML type azure, okta, onelogin or keycloakstring
lenses.security.saml.keyStoreFileDataThe base64 encoded contents of the keystorestring
lenses.security.saml.keyStorePasswordThe keystore passwordstring
lenses.security.saml.keyPasswordThe password for the SAML key within the keystorestring
lenses.security.saml.metadataFileDataThe base64 encoded content of the XML file provided by the Identity Providerstring

Example:

lenses:
  security:
     saml:
      enabled: true
      baseUrl: "https://lenses-prod.eastus2.cloudapp.azure.com"
      provider: "azure"
      keyStoreFileData: |-
                YmFzZTY0IG9mIGtleXN0b3JlIA==
      keyStorePassword: "password"
      keyPassword: "password"
      metadataFileData: |-
                LUlkUCBiYXNlNjQgWE1MIGZpbGUgY29udGVudC0=

Default admin user 

The security user contains the default username and password for the super administrator. See how to secure your admin account .

KeyDescriptionTypeDefaultRequired
lenses.security.userDefault admin usernamestringadminno
lenses.security.passwordDefault admin passwordstringadminno

Example:

lenses:
  security:
    defaultUser:
      username: admin
      password: admin

Sidecar containers 

With sidecar containers configuration you may deploy supporting containers such as data extractors/generators, etc. alongside Lenses container. You can read more about sidecars .

# Simple example
sidecarContainers:
   - name: sidecar-example
     image: alpine
     command: ["sh", "-c", "watch datetime"]

Running Kafka, ZK, KC on Kubernetes 

This allows the pods to have stable network identifiers. Each pods address should be added as an entry. The address takes the form of:

<statefulset-name-<pod ordinal identifier.<servicename.<namespace.svc.cluster.local

For example, if using statefulset of replicas called with a headless service the addresses would be:

    zookeeper-0.zookeeper.defaut.svc.cluster.local
    zookeeper-1.zookeeper.defaut.svc.cluster.local
    zookeeper-2.zookeeper.defaut.svc.cluster.local
    schema-registry-0.schema.default.svc.cluster.local
    schema-registry-1.schema.default.svc.cluster.local
    connect-worker-0.connect.defaut.svc.cluster.local
    connect-worker-1.connect.defaut.svc.cluster.local

If you only have one instance of a service you can set the service name.