Configuration Reference

This page lists the available configurations in Lenses.

Basics

Reference documentation of all configuration and authentication options:

Set in lenses.conf

Key Description Default Type Required

Key	Description	Default	Type	Required
lenses.ip	Bind HTTP at the given endpoint. Use in conjunction with `lenses.port`	0.0.0.0	string	no
lenses.port	The HTTP port to listen for API, UI and WS calls	9991	int	no
lenses.jmx.port	Bind JMX port to enable monitoring Lenses		int	no
lenses.root.path	The path from which all the Lenses URLs are served		string	no
lenses.secret.file	The full path to `security.conf` for security credentials	security.conf	string	no
lenses.sql.execution.mode	Streaming SQL mode `IN_PROC` (test mode) or `KUBERNETES` (prod mode)	IN_PROC	string	no
lenses.offset.workers	Number of workers to monitor topic offsets	5	int	no
lenses.telemetry.enable	Enable telemetry data collection	true	boolean	no
lenses.kafka.control.topics	An array of topics to be treated as “system topics”	list	array	no
lenses.grafana	Add your Grafana url i.e. http://grafanahost:port		string	no
lenses.api.response.cache.enable	If enabled, it disables client cache on the Lenses API HTTP responses by adding these HTTP Headers: `Cache-Control: no-cache, no-store, must-revalidate`, `Pragma: no-cache`, and `Expires: -1`.	false	boolean	no
lenses.workspace	Directory to write temp files. If write access is denied, Lenses will fallback to `/tmp`.	/run	string	no

lenses.ip

Bind HTTP at the given endpoint. Use in conjunction with lenses.port

0.0.0.0

string

lenses.port

The HTTP port to listen for API, UI and WS calls

9991

int

lenses.jmx.port

Bind JMX port to enable monitoring Lenses

int

lenses.root.path

The path from which all the Lenses URLs are served

string

lenses.secret.file

The full path to security.conf for security credentials

security.conf

string

lenses.sql.execution.mode

Streaming SQL mode IN_PROC (test mode) or KUBERNETES (prod mode)

IN_PROC

string

lenses.offset.workers

Number of workers to monitor topic offsets

int

lenses.telemetry.enable

Enable telemetry data collection

true

boolean

lenses.kafka.control.topics

An array of topics to be treated as “system topics”

list

array

lenses.grafana

Add your Grafana url i.e. http://grafanahost:port

string

lenses.api.response.cache.enable

If enabled, it disables client cache on the Lenses API HTTP responses by adding these HTTP Headers: Cache-Control: no-cache, no-store, must-revalidate, Pragma: no-cache, and Expires: -1.

false

boolean

lenses.workspace

Directory to write temp files. If write access is denied, Lenses will fallback to /tmp.

/run

string

Default system topics

System or control topics are created by services for their internal use. Below is the list of built-in configurations to identify them.

_schemas
__consumer_offsets
_kafka_lenses_
lsql_*
lsql-*
__transaction_state
__topology
__topology__metrics
_confluent*
*-KSTREAM-*
*-TableSource-*
*-changelog
__amazon_msk*

Wildcard (*) is used to match any name in the path to capture a list of topics not just one. When the wildcard is not specified, Lenses matches on the entry name provided.

Security

Set in security.conf

TLS

Key Description Default

Key	Description	Default
lenses.access.control.allow.methods	HTTP verbs allowed in cross-origin HTTP requests	`GET,POST,PUT,DELETE,OPTIONS`
lenses.access.control.allow.origin	Allowed hosts for cross-origin HTTP requests	`*`
lenses.allow.weak.ssl	Allow `https://` with self-signed certificates	`false`
lenses.ssl.keystore.location	The full path to the keystore file used to enable TLS on Lenses port
lenses.ssl.keystore.password	Password for the keystore file
lenses.ssl.key.password	Password for the ssl certificate used
lenses.ssl.enabled.protocols	Version of TLS protocol to use	`TLSv1.2`
lenses.ssl.algorithm	X509 or PKIX algorithm to use for TLS termination	`SunX509`
lenses.ssl.cipher.suites	Comma separated list of ciphers allowed for TLS negotiation

lenses.access.control.allow.methods

HTTP verbs allowed in cross-origin HTTP requests

GET,POST,PUT,DELETE,OPTIONS

lenses.access.control.allow.origin

Allowed hosts for cross-origin HTTP requests

*

lenses.allow.weak.ssl

Allow https:// with self-signed certificates

false

lenses.ssl.keystore.location

The full path to the keystore file used to enable TLS on Lenses port

lenses.ssl.keystore.password

Password for the keystore file

lenses.ssl.key.password

Password for the ssl certificate used

lenses.ssl.enabled.protocols

Version of TLS protocol to use

TLSv1.2

lenses.ssl.algorithm

X509 or PKIX algorithm to use for TLS termination

SunX509

lenses.ssl.cipher.suites

Comma separated list of ciphers allowed for TLS negotiation

LDAP

LDAP or AD connectivity is optional. All settings are string.

Set in security.conf

Key Description Default

Key	Description	Default
lenses.security.ldap.url	LDAP server URL (TLS, StartTLS and unencrypted supported)
lenses.security.ldap.user	LDAP user account. Must be able to list users and their groups. The distinguished name (DN) must be used
lenses.security.ldap.password	LDAP account password
lenses.security.ldap.base	LDAP base path for querying user accounts. All user accounts that will be able to access Lenses should be under this path
lenses.security.ldap.filter	LDAP query filter for matching users. Lenses will request all entries under the base path that satisfy this filter. The result should be unique	`(&(objectClass=person)(sAMAccountName=<user>))`
lenses.security.ldap.plugin.class	Full classpath that implements the LDAP query for the user’s groups. You can use the implementation that comes with Lenses if your LDAP setup is supported
lenses.security.ldap.plugin.memberof.key	LDAP user attribute that provides memberOf information. In most implementations the attribute has the same name, so you don’t have to set anything. Used by the default plugin	`memberOf`
lenses.security.ldap.plugin.group.extract.regex	A regular expression to extract a part of the user’s groups. If this part matches a Lenses group, the user will be granted all the permissions of this group. Lenses checks against the list of memberOf attribute values and uses the first regex group that is returned	`(?i)CN=(\\w+),ou=Groups.*`
lenses.security.ldap.plugin.person.name.key	This key is used by the included LDAP plugin class LdapMemberOfUserGroupPlugin. It expects the LDAP user attribute that provides the full name of the user	`sn`

lenses.security.ldap.url

LDAP server URL (TLS, StartTLS and unencrypted supported)

lenses.security.ldap.user

LDAP user account. Must be able to list users and their groups. The distinguished name (DN) must be used

lenses.security.ldap.password

LDAP account password

lenses.security.ldap.base

LDAP base path for querying user accounts. All user accounts that will be able to access Lenses should be under this path

lenses.security.ldap.filter

LDAP query filter for matching users. Lenses will request all entries under the base path that satisfy this filter. The result should be unique

(&(objectClass=person)(sAMAccountName=<user>))

lenses.security.ldap.plugin.class

Full classpath that implements the LDAP query for the user’s groups. You can use the implementation that comes with Lenses if your LDAP setup is supported

lenses.security.ldap.plugin.memberof.key

LDAP user attribute that provides memberOf information. In most implementations the attribute has the same name, so you don’t have to set anything. Used by the default plugin

memberOf

lenses.security.ldap.plugin.group.extract.regex

A regular expression to extract a part of the user’s groups. If this part matches a Lenses group, the user will be granted all the permissions of this group. Lenses checks against the list of memberOf attribute values and uses the first regex group that is returned

(?i)CN=(\\w+),ou=Groups.*

lenses.security.ldap.plugin.person.name.key

This key is used by the included LDAP plugin class LdapMemberOfUserGroupPlugin. It expects the LDAP user attribute that provides the full name of the user

sn

An additional configuration setting lenses.security.ldap.use.service.user.search when set to true will use the lenses.security.ldap.user account to read the groups of the currently logged user. The default behaviour (false) uses the currently logged user to read group memberships.

SSO SAML

Set in security.conf

Key Description Default

Key	Description	Default
lenses.security.saml.base.url	Lenses HTTPS URL that matches the Service Provider (SP) and part of the Identity Provider (IdP) SAML handshake i.e. `https://lenses-dev.example.com`
lenses.security.saml.sp.entityid	SAML Service Provider (SP) Entity ID for Lenses, used as part of the SAML handshake protocol.
lenses.security.saml.idp.provider	The Identity Provider (IdP) type: `azure`, `google`, `keycloak`, `okta`, `onelogin`
lenses.security.saml.idp.metadata.file	Path to XML file provided by the Identity Provider. e.g. /path/to/saml-idp.xml
lenses.security.saml.idp.session.lifetime.max	The maximum “duration since login” to accept from IdP. A SAML safety measure that is usually not used. See the duration syntax.	`100days`
lenses.security.saml.keystore.location	Location for the Java keystore file to be used for SAML crypto i.e. `/path/to/keystore.jks`
lenses.security.saml.keystore.password	Password for accessing the keystore
lenses.security.saml.key.alias	Alias to use for the private key within the keystore (only required when the keystore has multiple keys)
lenses.security.saml.key.password	Password for accessing the private key within the keystore

lenses.security.saml.base.url

Lenses HTTPS URL that matches the Service Provider (SP) and part of the Identity Provider (IdP) SAML handshake i.e. https://lenses-dev.example.com

lenses.security.saml.sp.entityid

SAML Service Provider (SP) Entity ID for Lenses, used as part of the SAML handshake protocol.

lenses.security.saml.idp.provider

The Identity Provider (IdP) type: azure, google, keycloak, okta, onelogin

lenses.security.saml.idp.metadata.file

Path to XML file provided by the Identity Provider. e.g. /path/to/saml-idp.xml

lenses.security.saml.idp.session.lifetime.max

The maximum “duration since login” to accept from IdP. A SAML safety measure that is usually not used. See the duration syntax.

100days

lenses.security.saml.keystore.location

Location for the Java keystore file to be used for SAML crypto i.e. /path/to/keystore.jks

lenses.security.saml.keystore.password

Password for accessing the keystore

lenses.security.saml.key.alias

Alias to use for the private key within the keystore (only required when the keystore has multiple keys)

lenses.security.saml.key.password

Password for accessing the private key within the keystore

Kerberos

Key Description Default

Key	Description	Default
lenses.security.kerberos.service.principal	The Kerberos principal for Lenses to use in the SPNEGO form: `HTTP/lenses.address@REALM.COM`
lenses.security.kerberos.keytab	Path to Kerberos keytab with the service principal. It should not be password protected
lenses.security.kerberos.debug	Enable Java’s JAAS debugging information	`false`

lenses.security.kerberos.service.principal

The Kerberos principal for Lenses to use in the SPNEGO form: HTTP/lenses.address@REALM.COM

lenses.security.kerberos.keytab

Path to Kerberos keytab with the service principal. It should not be password protected

lenses.security.kerberos.debug

Enable Java’s JAAS debugging information

false

Persistent storage

Common

Key	Description	Default	Type	Required
lenses.storage.hikaricp.[*]	To pass additional properties to HikariCP connection pool			no

Key

Description

Default

Type

Required

lenses.storage.hikaricp.[*]

To pass additional properties to HikariCP connection pool

H2

Key Description Default Type Required

Key	Description	Default	Type	Required
lenses.storage.directory	The full path to a directory for Lenses to use for persistence	`"./storage"`	string	no

lenses.storage.directory

The full path to a directory for Lenses to use for persistence

"./storage"

string

Postgres

Set in security.conf

Key Description Default Type Required

Key	Description	Default	Type	Required
lenses.storage.postgres.host	Host of PostgreSQL server for Lenses to use for persistence		string	no
lenses.storage.postgres.port	Port of PostgreSQL server for Lenses to use for persistence	`5432`	integer	no
lenses.storage.postgres.username	Username for PostgreSQL database user		string	no
lenses.storage.postgres.password	Password for PostgreSQL database user		string	no
lenses.storage.postgres.database	PostgreSQL database name for Lenses to use for persistence		string	no
lenses.storage.postgres.schema	PostgreSQL schema name for Lenses to use for persistence	`"public"`	string	no
lenses.storage.postgres.properties.[*]	To pass additional properties to PostgreSQL JDBC driver			no

lenses.storage.postgres.host

Host of PostgreSQL server for Lenses to use for persistence

string

lenses.storage.postgres.port

Port of PostgreSQL server for Lenses to use for persistence

5432

integer

lenses.storage.postgres.username

Username for PostgreSQL database user

string

lenses.storage.postgres.password

Password for PostgreSQL database user

string

lenses.storage.postgres.database

PostgreSQL database name for Lenses to use for persistence

string

lenses.storage.postgres.schema

PostgreSQL schema name for Lenses to use for persistence

"public"

string

lenses.storage.postgres.properties.[*]

To pass additional properties to PostgreSQL JDBC driver

Schema registries

Set in lenses.conf

If the records schema is centralized, the connectivity to Schema Registry nodes is defined by a Lenses Connection.

There are two static config entries to enable/disable the deletion of schemas:

Key Description Type

Key	Description	Type
lenses.schema.registry.delete	Allow schemas to be deleted. Default is `false`	boolean
lenses.schema.registry.cascade.delete	Deletes associated schemas when a topic is deleted. Default is `false`	boolean

lenses.schema.registry.delete

Allow schemas to be deleted. Default is false

boolean

lenses.schema.registry.cascade.delete

Deletes associated schemas when a topic is deleted. Default is false

boolean

Deployments

Set in lenses.conf

Options for specific deployment targets:

Global options
Kubernetes

Global options

Common settings, independently of the underlying deployment target:

Key Description Default

Key	Description	Default
lenses.deployments.events.buffer.size	Buffer size for events coming from Deployment targets such as Kubernetes	`10000`
lenses.deployments.errors.buffer.size	Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes	`1000`

lenses.deployments.events.buffer.size

Buffer size for events coming from Deployment targets such as Kubernetes

10000

lenses.deployments.errors.buffer.size

Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes

1000

Kubernetes

Kubernetes connectivity is optional. Minimum supported K8 version 0.11.10. All settings are string.

Set in lenses.conf

Key Description Default

Key	Description	Default
lenses.kubernetes.processor.image.name	The url for the streaming SQL Docker for K8	`lensesioextra/sql-processor`
lenses.kubernetes.processor.image.tag	The version/tag of the above container	`5.2`
lenses.kubernetes.config.file	The path for the `kubectrl` config file	`/home/lenses/.kube/config`
lenses.kubernetes.pull.policy	Pull policy for K8 containers: `IfNotPresent` or `Always`	`IfNotPresent`
lenses.kubernetes.service.account	The service account for deployments. Will also pull the image	`default`
lenses.kubernetes.init.container.image.name	The docker/container repository url and name of the Init Container image used to deploy applications to Kubernetes	`lensesio/lenses-cli`
lenses.kubernetes.init.container.image.tag	The tag of the Init Container image used to deploy applications to Kubernetes	`5.2.0`
lenses.kubernetes.watch.reconnect.limit	How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable	`10`
lenses.kubernetes.watch.reconnect.interval	How often to wait between Kubernetes Watcher reconnection attempts expressed in milliseconds	`5000`
lenses.kubernetes.websocket.timeout	How long to wait for a Kubernetes Websocket response expressed in milliseconds	`15000`
lenses.kubernetes.websocket.ping.interval	How often to ping Kubernetes Websocket to check it’s alive expressed in milliseconds	`30000`
lenses.kubernetes.pod.heap	The max amount of memory the underlying Java process will use	`900M`
lenses.kubernetes.pod.min.heap	The initial amount of memory the underlying Java process will allocate	`128M`
lenses.kubernetes.pod.mem.request	The value will control how much memory resource the Pod Container will request	`128M`
lenses.kubernetes.pod.mem.limit	The value will control the Pod Container memory limit	`1152M`
lenses.kubernetes.pod.cpu.request	The value will control how much cpu resource the Pod Container will request	`null`
lenses.kubernetes.pod.cpu.limit	The value will control the Pod Container cpu limit	`null`
lenses.kubernetes.namespaces	Object setting a list of Kubernetes namespaces that Lenses will see for each of the specified and configured cluster	`null`
lenses.kubernetes.pod.liveness.initial.delay	Amount of time Kubernetes will wait to check Processor’s health for the first time. It can be expressed like 30 second, 2 minute or 3 hour, mind the time unit is singular	`60 second`
lenses.deployments.events.buffer.size	Buffer size for events coming from Deployment targets such as Kubernetes	`10000`
lenses.deployments.errors.buffer.size	Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes	`1000`
lenses.kubernetes.config.reload.interval	Time interval to reload the Kubernetes configuration file. Expressed in milliseconds.	`30000`

lenses.kubernetes.processor.image.name

The url for the streaming SQL Docker for K8

lensesioextra/sql-processor

lenses.kubernetes.processor.image.tag

The version/tag of the above container

5.2

lenses.kubernetes.config.file

The path for the kubectrl config file

/home/lenses/.kube/config

lenses.kubernetes.pull.policy

Pull policy for K8 containers: IfNotPresent or Always

IfNotPresent

lenses.kubernetes.service.account

The service account for deployments. Will also pull the image

default

lenses.kubernetes.init.container.image.name

The docker/container repository url and name of the Init Container image used to deploy applications to Kubernetes

lensesio/lenses-cli

lenses.kubernetes.init.container.image.tag

The tag of the Init Container image used to deploy applications to Kubernetes

5.2.0

lenses.kubernetes.watch.reconnect.limit

How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable

10

lenses.kubernetes.watch.reconnect.interval

How often to wait between Kubernetes Watcher reconnection attempts expressed in milliseconds

5000

lenses.kubernetes.websocket.timeout

How long to wait for a Kubernetes Websocket response expressed in milliseconds

15000

lenses.kubernetes.websocket.ping.interval

How often to ping Kubernetes Websocket to check it’s alive expressed in milliseconds

30000

lenses.kubernetes.pod.heap

The max amount of memory the underlying Java process will use

900M

lenses.kubernetes.pod.min.heap

The initial amount of memory the underlying Java process will allocate

128M

lenses.kubernetes.pod.mem.request

The value will control how much memory resource the Pod Container will request

128M

lenses.kubernetes.pod.mem.limit

The value will control the Pod Container memory limit

1152M

lenses.kubernetes.pod.cpu.request

The value will control how much cpu resource the Pod Container will request

null

lenses.kubernetes.pod.cpu.limit

The value will control the Pod Container cpu limit

null

lenses.kubernetes.namespaces

Object setting a list of Kubernetes namespaces that Lenses will see for each of the specified and configured cluster

null

lenses.kubernetes.pod.liveness.initial.delay

Amount of time Kubernetes will wait to check Processor’s health for the first time. It can be expressed like 30 second, 2 minute or 3 hour, mind the time unit is singular

60 second

lenses.deployments.events.buffer.size

Buffer size for events coming from Deployment targets such as Kubernetes

10000

lenses.deployments.errors.buffer.size

Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes

1000

lenses.kubernetes.config.reload.interval

Time interval to reload the Kubernetes configuration file. Expressed in milliseconds.

30000

SQL snapshot (Explore & Studio)

Optimization settings for SQL queries.

Set in lenses.conf

Key Description Type Default

Key	Description	Type	Default
lenses.sql.settings.max.size	Restricts the max bytes that a kafka sql query will return	long	`20971520` (20MB)
lenses.sql.settings.max.query.time	Max time (in msec) that a sql query will run	int	`3600000` (1h)
lenses.sql.settings.max.idle.time	Max time (in msec) for a query when it reaches the end of the topic	int	`5000` (5 sec)
lenses.sql.settings.show.bad.records	By default show bad records when querying a kafka topic	boolean	`true`
lenses.sql.settings.format.timestamp	By default convert AVRO date to human readable format	boolean	`true`
lenses.sql.settings.live.aggs	By default allow aggregation queries on kafka data	boolean	`true`
lenses.sql.sample.default	Number of messages to sample when live tailing a kafka topic	int	`2`/window
lenses.sql.sample.window	How frequently to sample messages when tailing a kafka topic	int	`200` msec
lenses.sql.websocket.buffer	Buffer size for messages in a SQL query	int	`10000`
lenses.metrics.workers	Number of workers for parallelising SQL queries	int	`16`
lenses.kafka.ws.buffer.size	Buffer size for WebSocket consumer	int	`10000`
lenses.kafka.ws.max.poll.records	Max number of kafka messages to return in a single poll()	long	`1000`
lenses.sql.state.dir	Folder to store KStreams state.	string	`logs/lenses-sql-kstream-state`
lenses.sql.udf.packages	The list of allowed java packages for UDFs/UDAFs	array of strings	`["io.lenses.sql.udf"]`

lenses.sql.settings.max.size

Restricts the max bytes that a kafka sql query will return

long

20971520 (20MB)

lenses.sql.settings.max.query.time

Max time (in msec) that a sql query will run

int

3600000 (1h)

lenses.sql.settings.max.idle.time

Max time (in msec) for a query when it reaches the end of the topic

int

5000 (5 sec)

lenses.sql.settings.show.bad.records

By default show bad records when querying a kafka topic

boolean

true

lenses.sql.settings.format.timestamp

By default convert AVRO date to human readable format

boolean

true

lenses.sql.settings.live.aggs

By default allow aggregation queries on kafka data

boolean

true

lenses.sql.sample.default

Number of messages to sample when live tailing a kafka topic

int

2/window

lenses.sql.sample.window

How frequently to sample messages when tailing a kafka topic

int

200 msec

lenses.sql.websocket.buffer

Buffer size for messages in a SQL query

int

10000

lenses.metrics.workers

Number of workers for parallelising SQL queries

int

16

lenses.kafka.ws.buffer.size

Buffer size for WebSocket consumer

int

10000

lenses.kafka.ws.max.poll.records

Max number of kafka messages to return in a single poll()

long

1000

lenses.sql.state.dir

Folder to store KStreams state.

string

logs/lenses-sql-kstream-state

lenses.sql.udf.packages

The list of allowed java packages for UDFs/UDAFs

array of strings

["io.lenses.sql.udf"]

Lenses internal Kafka topics

Lenses requires these Kafka topics to be available, otherwise, it will try to create them. The topics can be created manually before Lenses is run, or allow Lenses the correct Kafka ACLs to create the topics:

Set in lenses.conf

Key Description Partition Replication Default Compacted Retention

Key	Description	Partition	Replication	Default	Compacted	Retention
lenses.topics.external.topology	Topic for applications to publish their topology	`1`	`3` (recommended)	`__topology`	yes	N/A
lenses.topics.external.metrics	Topic for external application to publish their metrics	`1`	`3` (recommended)	`__topology__metrics`	no	1 day
lenses.topics.metrics	Topic for SQL Processor to send the metrics	`1`	`3` (recommended)	`_kafka_lenses_metrics`	no

lenses.topics.external.topology

Topic for applications to publish their topology

1

3 (recommended)

__topology

yes

N/A

lenses.topics.external.metrics

Topic for external application to publish their metrics

1

3 (recommended)

__topology__metrics

1 day

lenses.topics.metrics

Topic for SQL Processor to send the metrics

1

3 (recommended)

_kafka_lenses_metrics

To allow for fine-grained control over the replication factor of the three topics, the following settings are available:

Key Description Default

Key	Description	Default
lenses.topics.replication.external.topology	Replication factor for the `lenses.topics.external.topology` topic	1
lenses.topics.replication.external.metrics	Replication factor for the `lenses.topics.external.metrics` topic	1
lenses.topics.replication.metrics	Replication factor for the `lenses.topics.metrics` topic	1

lenses.topics.replication.external.topology

Replication factor for the lenses.topics.external.topology topic

lenses.topics.replication.external.metrics

Replication factor for the lenses.topics.external.metrics topic

lenses.topics.replication.metrics

Replication factor for the lenses.topics.metrics topic

When configuring the replication factor for your deployment, it's essential to consider the requirements imposed by your cloud provider. Many cloud providers enforce a minimum replication factor to ensure data durability and high availability. For example, IBM Cloud mandates a minimum replication factor of 3. Therefore, it's crucial to set the replication factor for the Lenses internal topics to at least 3 when deploying Lenses on IBM Cloud.

Advanced

All time configuration options are in milliseconds.

Set in lenses.conf

Key Description Type Default

Key	Description	Type	Default
lenses.interval.summary	How often to refresh kafka topic list and configs	long	`10000`
lenses.interval.consumers.refresh.ms	How often to refresh kafka consumer group info	long	`10000`
lenses.interval.consumers.timeout.ms	How long to wait for kafka consumer group info to be retrieved	long	`300000`
lenses.interval.partitions.messages	How often to refresh kafka partition info	long	`10000`
lenses.interval.type.detection	How often to check kafka topic payload info	long	`30000`
lenses.interval.user.session.ms	How long a client-session stays alive if inactive (4 hours)	long	`14400000`
lenses.interval.user.session.refresh	How often to check for idle client sessions	long	`60000`
lenses.interval.topology.topics.metrics	How often to refresh topology info	long	`30000`
lenses.interval.schema.registry.healthcheck	How often to check the schema registries health	long	`30000`
lenses.interval.schema.registry.refresh.ms	How often to refresh schema registry data	long	`30000`
lenses.interval.metrics.refresh.zk	How often to refresh ZK metrics	long	`5000`
lenses.interval.metrics.refresh.sr	How often to refresh Schema Registry metrics	long	`5000`
lenses.interval.metrics.refresh.broker	How often to refresh Kafka Broker metrics	long	`5000`
lenses.interval.metrics.refresh.connect	How often to refresh Kafka Connect metrics	long	`30000`
lenses.interval.metrics.refresh.brokers.in.zk	How often to refresh from ZK the Kafka broker list	long	`5000`
lenses.interval.topology.timeout.ms	Time period when a metric is considered stale	long	`120000`
lenses.interval.audit.data.cleanup	How often to clean up dataset view entries from the audit log	long	`300000`
lenses.audit.to.log.file	Path to a file to write audits to in JSON format.	string
lenses.interval.jmxcache.refresh.ms	How often to refresh the JMX cache used in the Explore page	long	`180000`
lenses.interval.jmxcache.graceperiod.ms	How long to pause for when a JMX connectity error occurs	long	`300000`
lenses.interval.jmxcache.timeout.ms	How long to wait for a JMX response	long	`500`
lenses.interval.sql.udf	How often to look for new UDF/UDAF (user defined [aggregate] functions)	long	`10000`
lenses.kafka.consumers.batch.size	How many consumer groups to retrieve in a single request	Int	`500`
lenses.kafka.ws.heartbeat.ms	How often to send heartbeat messages in TCP connection	long	`30000`
lenses.kafka.ws.poll.ms	Max time for kafka consumer data polling on WS APIs	long	`10000`
lenses.kubernetes.config.reload.interval	Time interval to reload the Kubernetes configuration file.	long	`30000`
lenses.kubernetes.watch.reconnect.limit	How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable	long	`10`
lenses.kubernetes.watch.reconnect.interval	How often to wait between Kubernetes Watcher reconnection attempts	long	`5000`
lenses.kubernetes.websocket.timeout	How long to wait for a Kubernetes Websocket response	long	`15000`
lenses.kubernetes.websocket.ping.interval	How often to ping Kubernetes Websocket to check it’s alive	long	`30000`
lenses.akka.request.timeout.ms	Max time for a response in an Akka Actor	long	`10000`
lenses.sql.monitor.frequency	How often to emit healthcheck and performance metrics on Streaming SQL	long	`10000`
lenses.audit.data.access	Record dataset access as audit log entries	boolean	`true`
lenses.audit.data.max.records	How many dataset view entries to retain in the audit log. Set to `-1` to retain indefinitely	int	`500000`
lenses.explore.lucene.max.clause.count	Override Lucene’s maximum number of clauses permitted per BooleanQuery	int	`1024`
lenses.explore.queue.size	Optional setting to bound Lenses internal queue used by the catalog subsystem. It needs to be positive integer or it will be ignored.	int	N/A
lenses.interval.kafka.connect.http.timeout.ms	How long to wait for Kafka Connect response to be retrieved	int	`10000`
lenses.interval.kafka.connect.healthcheck	How often to check the Kafka health	int	`15000`
lenses.interval.schema.registry.http.timeout.ms	How long to wait for Schema Registry response to be retrieved	int	`10000`
lenses.interval.zookeeper.healthcheck	How often to check the Zookeeper health	int	`15000`
lenses.ui.topics.row.limit	The number of Kafka records to load automatically when exploring a topic	int	`200`
lenses.deployments.connect.failure.alert.check.interval	Time interval in seconds to check the connector failure grace period has completed. Used by the Connect auto-restart failed connectors functionality. It needs too be a value between (1,600].	int	`10`
lenses.provisioning.path	Folder on the filesystem containing the provisioning data. See [provisioning docs](link to provisioning docs) for further details	string
lenses.provisioning.interval	Time interval in seconds to check for changes on the provisioning resources	int
lenses.schema.registry.client.http.retryOnTooManyRequest	When enabled, Lenses will retry a request whenever the schema registry returns a `429 Too Many Requests`	boolean	`false`
lenses.schema.registry.client.http.maxRetryAwait	Max amount of time to wait whenever a `429 Too Many Requests` is returned.	duration	`"2 seconds"`
lenses.schema.registry.client.http.maxRetryCount	Max retry count whenever a `429 Too Many Requests` is returned.	integer	2
lenses.schema.registry.client.http.rate.type	Specifies if http requests to the configured schema registry should be rate limited. Can be "session" or "unlimited"	"unlimited" \| "session"	`unlimited`
lenses.schema.registry.client.http.rate.maxRequests	Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.	integer	N/A
lenses.schema.registry.client.http.rate.window	Whenever the rate limiter is "session" this configuration will determine the duration of the window used.	duration	N/A
lenses.schema.connect.client.http.retryOnTooManyRequest	Retry a request whenever a connect cluster returns a 429 `Too Many Requests`	boolean	`false`
lenses.schema.connect.client.http.maxRetryAwait	Max amount of time to wait whenever a `429 Too Many Requests` is returned.	duration	`2 seconds`
lenses.schema.connect.client.http.maxRetryCount	Max retry count whenever a `429 Too Many Requests` is returned.	integer	2
lenses.connect.client.http.rate.type	Specifies if http requests to the configured connect cluster should be rate limited. Can be "session" or "unlimited"	"unlimited" \| "session"	`unlimited`
lenses.connect.client.http.rate.maxRequests	Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.	integer	N/A
lenses.connect.client.http.rate.window	Whenever the rate limiter is "session" this configuration will determine the duration of the window used.	duration	N/A

lenses.interval.summary

How often to refresh kafka topic list and configs

long

10000

lenses.interval.consumers.refresh.ms

How often to refresh kafka consumer group info

long

10000

lenses.interval.consumers.timeout.ms

How long to wait for kafka consumer group info to be retrieved

long

300000

lenses.interval.partitions.messages

How often to refresh kafka partition info

long

10000

lenses.interval.type.detection

How often to check kafka topic payload info

long

30000

lenses.interval.user.session.ms

How long a client-session stays alive if inactive (4 hours)

long

14400000

lenses.interval.user.session.refresh

How often to check for idle client sessions

long

60000

lenses.interval.topology.topics.metrics

How often to refresh topology info

long

30000

lenses.interval.schema.registry.healthcheck

How often to check the schema registries health

long

30000

lenses.interval.schema.registry.refresh.ms

How often to refresh schema registry data

long

30000

lenses.interval.metrics.refresh.zk

How often to refresh ZK metrics

long

5000

lenses.interval.metrics.refresh.sr

How often to refresh Schema Registry metrics

long

5000

lenses.interval.metrics.refresh.broker

How often to refresh Kafka Broker metrics

long

5000

lenses.interval.metrics.refresh.connect

How often to refresh Kafka Connect metrics

long

30000

lenses.interval.metrics.refresh.brokers.in.zk

How often to refresh from ZK the Kafka broker list

long

5000

lenses.interval.topology.timeout.ms

Time period when a metric is considered stale

long

120000

lenses.interval.audit.data.cleanup

How often to clean up dataset view entries from the audit log

long

300000

lenses.audit.to.log.file

Path to a file to write audits to in JSON format.

string

lenses.interval.jmxcache.refresh.ms

How often to refresh the JMX cache used in the Explore page

long

180000

lenses.interval.jmxcache.graceperiod.ms

How long to pause for when a JMX connectity error occurs

long

300000

lenses.interval.jmxcache.timeout.ms

How long to wait for a JMX response

long

500

lenses.interval.sql.udf

How often to look for new UDF/UDAF (user defined [aggregate] functions)

long

10000

lenses.kafka.consumers.batch.size

How many consumer groups to retrieve in a single request

Int

500

lenses.kafka.ws.heartbeat.ms

How often to send heartbeat messages in TCP connection

long

30000

lenses.kafka.ws.poll.ms

Max time for kafka consumer data polling on WS APIs

long

10000

lenses.kubernetes.config.reload.interval

Time interval to reload the Kubernetes configuration file.

long

30000

lenses.kubernetes.watch.reconnect.limit

How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable

long

10

lenses.kubernetes.watch.reconnect.interval

How often to wait between Kubernetes Watcher reconnection attempts

long

5000

lenses.kubernetes.websocket.timeout

How long to wait for a Kubernetes Websocket response

long

15000

lenses.kubernetes.websocket.ping.interval

How often to ping Kubernetes Websocket to check it’s alive

long

30000

lenses.akka.request.timeout.ms

Max time for a response in an Akka Actor

long

10000

lenses.sql.monitor.frequency

How often to emit healthcheck and performance metrics on Streaming SQL

long

10000

lenses.audit.data.access

Record dataset access as audit log entries

boolean

true

lenses.audit.data.max.records

How many dataset view entries to retain in the audit log. Set to -1 to retain indefinitely

int

500000

lenses.explore.lucene.max.clause.count

Override Lucene’s maximum number of clauses permitted per BooleanQuery

int

1024

lenses.explore.queue.size

Optional setting to bound Lenses internal queue used by the catalog subsystem. It needs to be positive integer or it will be ignored.

int

N/A

lenses.interval.kafka.connect.http.timeout.ms

How long to wait for Kafka Connect response to be retrieved

int

10000

lenses.interval.kafka.connect.healthcheck

How often to check the Kafka health

int

15000

lenses.interval.schema.registry.http.timeout.ms

How long to wait for Schema Registry response to be retrieved

int

10000

lenses.interval.zookeeper.healthcheck

How often to check the Zookeeper health

int

15000

lenses.ui.topics.row.limit

The number of Kafka records to load automatically when exploring a topic

int

200

lenses.deployments.connect.failure.alert.check.interval

Time interval in seconds to check the connector failure grace period has completed. Used by the Connect auto-restart failed connectors functionality. It needs too be a value between (1,600].

int

10

lenses.provisioning.path

Folder on the filesystem containing the provisioning data. See [provisioning docs](link to provisioning docs) for further details

string

lenses.provisioning.interval

Time interval in seconds to check for changes on the provisioning resources

int

lenses.schema.registry.client.http.retryOnTooManyRequest

When enabled, Lenses will retry a request whenever the schema registry returns a 429 Too Many Requests

boolean

false

lenses.schema.registry.client.http.maxRetryAwait

Max amount of time to wait whenever a 429 Too Many Requests is returned.

duration

"2 seconds"

lenses.schema.registry.client.http.maxRetryCount

Max retry count whenever a 429 Too Many Requests is returned.

integer

lenses.schema.registry.client.http.rate.type

Specifies if http requests to the configured schema registry should be rate limited. Can be "session" or "unlimited"

"unlimited" | "session"

unlimited

lenses.schema.registry.client.http.rate.maxRequests

Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.

integer

N/A

lenses.schema.registry.client.http.rate.window

Whenever the rate limiter is "session" this configuration will determine the duration of the window used.

duration

N/A

lenses.schema.connect.client.http.retryOnTooManyRequest

Retry a request whenever a connect cluster returns a 429 Too Many Requests

boolean

false

lenses.schema.connect.client.http.maxRetryAwait

Max amount of time to wait whenever a 429 Too Many Requests is returned.

duration

2 seconds

lenses.schema.connect.client.http.maxRetryCount

Max retry count whenever a 429 Too Many Requests is returned.

integer

lenses.connect.client.http.rate.type

Specifies if http requests to the configured connect cluster should be rate limited. Can be "session" or "unlimited"

"unlimited" | "session"

unlimited

lenses.connect.client.http.rate.maxRequests

Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.

integer

N/A

lenses.connect.client.http.rate.window

Whenever the rate limiter is "session" this configuration will determine the duration of the window used.

duration

N/A

Connectors topology

Set in lenses.conf

Control how Lenses identifies your connectors in the Topology view. Catalogue your connector types, set their icons, and control how Lenses extracts the topics used by your connectors.

Lenses comes preconfigured for some of the popular connectors as well as the Stream Reactor connectors. If you see that Lenses doesn’t automatically identify your connector type then use the lenses.connectors.info setting to register it with Lenses.

Add a new HOCON object {} for every new Connector in your lenses.connectors.info list :

  lenses.connectors.info = [
      {
        class.name = "The connector full classpath"
        name = "The name which will be presented in the UI"
        instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
        sink = true
        extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
        icon = "file.png"
        description = "A description for the connector"
        author = "The connector author"
      }
  ]

This configuration allows the connector to work with the topology graph, and also have the RBAC rules applied to it.

Source example

To extract the topic information from the connector configuration, source connectors require an extra configuration. The extractor class should be: io.lenses.config.kafka.connect.SimpleTopicsExtractor. Using this extractor requires an extra property configuration. It specifies the field in the connector configuration which determines the topics data is sent to.

Here is an example for the file source:

  lenses.connectors.info = [
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSource"
      name = "File"
      instance = "file"
      sink = false
      property = "topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
    }
  ]

Sink example

An example of a Splunk sink connector and a Debezium SQL server connector

  lenses.connectors.info = [
    {
      class.name = "com.splunk.kafka.connect.SplunkSinkConnector"
      name = "Splunk Sink",
      instance = "splunk.hec.uri"
      sink = true,
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "splunk.png",
      description = "Stores Kafka data in Splunk"
      docs = "https://github.com/splunk/kafka-connect-splunk",
      author = "Splunk"
    },
    {
      class.name = "io.debezium.connector.sqlserver.SqlServerConnector"
      name = "CDC MySQL"
      instance = "database.hostname"
      sink = false,
      property = "database.history.kafka.topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "debezium.png"
      description = "CDC data from RDBMS into Kafka"
      docs = "//debezium.io/docs/connectors/mysql/",
      author = "Debezium"
    }
  ]

External Applications

Set in lenses.conf

Key Description Default Type Required

Key	Description	Default	Type	Required
apps.external.http.state.refresh.ms	When registering a runner for external app, a health-check interval can be specified. If it is not, this default interval is used (value in milliseconds)	`30000`	int	no
apps.external.http.state.cache.expiration.ms	Last known state of the runner is stored in a cache. The entries in the cache are being invalidated after a time that is defined by following configuration key (value in milliseconds). This value should not be lower than the `apps.external.http.state.refresh.ms` value.	`60000`	int	no

apps.external.http.state.refresh.ms

When registering a runner for external app, a health-check interval can be specified. If it is not, this default interval is used (value in milliseconds)

30000

int

apps.external.http.state.cache.expiration.ms

Last known state of the runner is stored in a cache. The entries in the cache are being invalidated after a time that is defined by following configuration key (value in milliseconds). This value should not be lower than the apps.external.http.state.refresh.ms value.

60000

int

PreviousPlugins NextTopics

Last updated 1 month ago