Configuration Reference

This page lists the available configurations in Lenses Agent.

Set in lenses.conf

Basics

Reference documentation of all configuration and authentication options:

Key	Description	Default	Type	Required
lenses.eula.accept	Accept the Lenses EULA	false	boolean	yes
lenses.ip	Bind HTTP at the given endpoint. Use in conjunction with lenses.port	0.0.0.0	string	no
lenses.port	The HTTP port to listen for API, UI and WS calls	9991	int	no
lenses.jmx.port	Bind JMX port to enable monitoring Lenses		int	no
lenses.root.path	The path from which all the Lenses URLs are served		string	no
lenses.secret.file	The full path to security.conf for security credentials	security.conf	string	no
lenses.sql.execution.mode	Streaming SQL mode IN_PROC (test mode) or KUBERNETES (prod mode)	IN_PROC	string	no
lenses.offset.workers	Number of workers to monitor topic offsets	5	int	no
lenses.telemetry.enable	Enable telemetry data collection	true	boolean	no
lenses.kafka.control.topics	An array of topics to be treated as “system topics”	list	array	no
lenses.grafana	Add your Grafana url i.e. http://grafanahost:port		string	no
lenses.api.response.cache.enable	If enabled, it disables client cache on the Lenses API HTTP responses by adding these HTTP Headers: Cache-Control: no-cache, no-store, must-revalidate, Pragma: no-cache, and Expires: -1.	false	boolean	no
lenses.workspace	Directory to write temp files. If write access is denied, Lenses will fallback to /tmp.	/run	string	no

Default system topics

System or control topics are created by services for their internal use. Below is the list of built-in configurations to identify them.

• _schemas

• __consumer_offsets

• _kafka_lenses_

• lsql_*

• lsql-*

• __transaction_state

• __topology

• __topology__metrics

• _confluent*

• *-KSTREAM-*

• *-TableSource-*

• *-changelog

• __amazon_msk*

Wildcard (*) is used to match any name in the path to capture a list of topics not just one. When the wildcard is not specified, Lenses matches on the entry name provided.

Security

TLS

Key	Description	Default
lenses.access.control.allow.methods	HTTP verbs allowed in cross-origin HTTP requests	GET,POST,PUT,DELETE,OPTIONS
lenses.access.control.allow.origin	Allowed hosts for cross-origin HTTP requests	*
lenses.allow.weak.ssl	Allow https:// with self-signed certificates	false
lenses.ssl.keystore.location	The full path to the keystore file used to enable TLS on Lenses port
lenses.ssl.keystore.password	Password for the keystore file
lenses.ssl.key.password	Password for the ssl certificate used
lenses.ssl.enabled.protocols	Version of TLS protocol to use	TLSv1.2
lenses.ssl.algorithm	X509 or PKIX algorithm to use for TLS termination	SunX509
lenses.ssl.cipher.suites	Comma separated list of ciphers allowed for TLS negotiation

Kerberos

Key	Description	Default
lenses.security.kerberos.service.principal	The Kerberos principal for Lenses to use in the SPNEGO form: HTTP/[e-mail]
lenses.security.kerberos.keytab	Path to Kerberos keytab with the service principal. It should not be password protected
lenses.security.kerberos.debug	Enable Java’s JAAS debugging information	false

Persistent storage

---

Common

Key	Description	Default	Type	Required
lenses.storage.hikaricp.[*]	To pass additional properties to HikariCP connection pool			no

H2

Key	Description	Default	Type	Required
lenses.storage.directory	The full path to a directory for Lenses to use for persistence	"./storage"	string	no

Postgres

Key	Description	Default	Type	Required
lenses.storage.postgres.host	Host of PostgreSQL server for Lenses to use for persistence		string	no
lenses.storage.postgres.port	Port of PostgreSQL server for Lenses to use for persistence	5432	integer	no
lenses.storage.postgres.username	Username for PostgreSQL database user		string	no
lenses.storage.postgres.password	Password for PostgreSQL database user		string	no
lenses.storage.postgres.database	PostgreSQL database name for Lenses to use for persistence		string	no
lenses.storage.postgres.schema	PostgreSQL schema name for Lenses to use for persistence	"public"	string	no
lenses.storage.postgres.properties.[*]	To pass additional properties to PostgreSQL JDBC driver			no

Schema registries

If the records schema is centralized, the connectivity to Schema Registry nodes is defined by a Lenses Connection.

There are two static config entries to enable/disable the deletion of schemas:

Key	Description	Type
lenses.schema.registry.delete	Allow schemas to be deleted. Default is false	boolean
lenses.schema.registry.cascade.delete	Deletes associated schemas when a topic is deleted. Default is false	boolean

Deployments

Options for specific deployment targets:

• Global options

• Kubernetes

Global options

Common settings, independently of the underlying deployment target:

Key	Description	Default
lenses.deployments.events.buffer.size	Buffer size for events coming from Deployment targets such as Kubernetes	10000
lenses.deployments.errors.buffer.size	Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes	1000

Kubernetes

Kubernetes connectivity is optional. Minimum supported K8 version 0.11.10. All settings are string.

Key	Description	Default
lenses.kubernetes.processor.image.name	The url for the streaming SQL Docker for K8	lensesioextra/sql-processor
lenses.kubernetes.processor.image.tag	The version/tag of the above container	5.2
lenses.kubernetes.config.file	The path for the kubectrl config file	/home/lenses/.kube/config
lenses.kubernetes.pull.policy	Pull policy for K8 containers: IfNotPresent or Always	IfNotPresent
lenses.kubernetes.service.account	The service account for deployments. Will also pull the image	default
lenses.kubernetes.init.container.image.name	The docker/container repository url and name of the Init Container image used to deploy applications to Kubernetes	lensesio/lenses-cli
lenses.kubernetes.init.container.image.tag	The tag of the Init Container image used to deploy applications to Kubernetes	5.2.0
lenses.kubernetes.watch.reconnect.limit	How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable	10
lenses.kubernetes.watch.reconnect.interval	How often to wait between Kubernetes Watcher reconnection attempts expressed in milliseconds	5000
lenses.kubernetes.websocket.timeout	How long to wait for a Kubernetes Websocket response expressed in milliseconds	15000
lenses.kubernetes.websocket.ping.interval	How often to ping Kubernetes Websocket to check it’s alive expressed in milliseconds	30000
lenses.kubernetes.pod.heap	The max amount of memory the underlying Java process will use	900M
lenses.kubernetes.pod.min.heap	The initial amount of memory the underlying Java process will allocate	128M
lenses.kubernetes.pod.mem.request	The value will control how much memory resource the Pod Container will request	128M
lenses.kubernetes.pod.mem.limit	The value will control the Pod Container memory limit	1152M
lenses.kubernetes.pod.cpu.request	The value will control how much cpu resource the Pod Container will request	null
lenses.kubernetes.pod.cpu.limit	The value will control the Pod Container cpu limit	null
lenses.kubernetes.namespaces	Object setting a list of Kubernetes namespaces that Lenses will see for each of the specified and configured cluster	null
lenses.kubernetes.pod.liveness.initial.delay	Amount of time Kubernetes will wait to check Processor’s health for the first time. It can be expressed like 30 second, 2 minute or 3 hour, mind the time unit is singular	60 second
lenses.deployments.events.buffer.size	Buffer size for events coming from Deployment targets such as Kubernetes	10000
lenses.deployments.errors.buffer.size	Buffer size for errors happening on the communication between Lenses and the Deployment targets such as Kubernetes	1000
lenses.kubernetes.config.reload.interval	Time interval to reload the Kubernetes configuration file. Expressed in milliseconds.	30000

SQL snapshot (Explore & Studio)

Optimization settings for SQL queries.

Key	Description	Type	Default
lenses.sql.settings.max.size	Restricts the max bytes that a kafka sql query will return	long	20971520 (20MB)
lenses.sql.settings.max.query.time	Max time (in msec) that a sql query will run	int	3600000 (1h)
lenses.sql.settings.max.idle.time	Max time (in msec) for a query when it reaches the end of the topic	int	5000 (5 sec)
lenses.sql.settings.show.bad.records	By default show bad records when querying a kafka topic	boolean	true
lenses.sql.settings.format.timestamp	By default convert AVRO date to human readable format	boolean	true
lenses.sql.settings.live.aggs	By default allow aggregation queries on kafka data	boolean	true
lenses.sql.sample.default	Number of messages to sample when live tailing a kafka topic	int	2/window
lenses.sql.sample.window	How frequently to sample messages when tailing a kafka topic	int	200 msec
lenses.sql.websocket.buffer	Buffer size for messages in a SQL query	int	10000
lenses.metrics.workers	Number of workers for parallelising SQL queries	int	16
lenses.kafka.ws.buffer.size	Buffer size for WebSocket consumer	int	10000
lenses.kafka.ws.max.poll.records	Max number of kafka messages to return in a single poll()	long	1000
lenses.sql.state.dir	Folder to store KStreams state.	string	logs/sql-kstream-state
lenses.sql.udf.packages	The list of allowed java packages for UDFs/UDAFs	array of strings	["io.lenses.sql.udf"]

Lenses internal Kafka topics

Lenses requires these Kafka topics to be available, otherwise, it will try to create them. The topics can be created manually before Lenses is run, or allow Lenses the correct Kafka ACLs to create the topics:

Key	Description	Partition	Replication	Default	Compacted	Retention
lenses.topics.external.topology	Topic for applications to publish their topology	1	3 (recommended)	__topology	yes	N/A
lenses.topics.external.metrics	Topic for external application to publish their metrics	1	3 (recommended)	__topology__metrics	no	1 day
lenses.topics.metrics	Topic for SQL Processor to send the metrics	1	3 (recommended)	_kafka_lenses_metrics	no

To allow for fine-grained control over the replication factor of the three topics, the following settings are available:

Key	Description	Default
lenses.topics.replication.external.topology	Replication factor for the lenses.topics.external.topology topic	1
lenses.topics.replication.external.metrics	Replication factor for the lenses.topics.external.metrics topic	1
lenses.topics.replication.metrics	Replication factor for the lenses.topics.metrics topic	1

When configuring the replication factor for your deployment, it's essential to consider the requirements imposed by your cloud provider. Many cloud providers enforce a minimum replication factor to ensure data durability and high availability. For example, IBM Cloud mandates a minimum replication factor of 3. Therefore, it's crucial to set the replication factor for the Lenses internal topics to at least 3 when deploying Lenses on IBM Cloud.

Advanced

All time configuration options are in milliseconds.

Key	Description	Type	Default
lenses.interval.summary	How often to refresh kafka topic list and configs	long	10000
lenses.interval.consumers.refresh.ms	How often to refresh kafka consumer group info	long	10000
lenses.interval.consumers.timeout.ms	How long to wait for kafka consumer group info to be retrieved	long	300000
lenses.interval.partitions.messages	How often to refresh kafka partition info	long	10000
lenses.interval.type.detection	How often to check kafka topic payload info	long	30000
lenses.interval.user.session.ms	How long a client-session stays alive if inactive (4 hours)	long	14400000
lenses.interval.user.session.refresh	How often to check for idle client sessions	long	60000
lenses.interval.topology.topics.metrics	How often to refresh topology info	long	30000
lenses.interval.schema.registry.healthcheck	How often to check the schema registries health	long	30000
lenses.interval.schema.registry.refresh.ms	How often to refresh schema registry data	long	30000
lenses.interval.metrics.refresh.zk	How often to refresh ZK metrics	long	5000
lenses.interval.metrics.refresh.sr	How often to refresh Schema Registry metrics	long	5000
lenses.interval.metrics.refresh.broker	How often to refresh Kafka Broker metrics	long	5000
lenses.interval.metrics.refresh.connect	How often to refresh Kafka Connect metrics	long	30000
lenses.interval.metrics.refresh.brokers.in.zk	How often to refresh from ZK the Kafka broker list	long	5000
lenses.interval.topology.timeout.ms	Time period when a metric is considered stale	long	120000
lenses.interval.audit.data.cleanup	How often to clean up dataset view entries from the audit log	long	300000
lenses.audit.to.log.file	Path to a file to write audits to in JSON format.	string
lenses.interval.jmxcache.refresh.ms	How often to refresh the JMX cache used in the Explore page	long	180000
lenses.interval.jmxcache.graceperiod.ms	How long to pause for when a JMX connectity error occurs	long	300000
lenses.interval.jmxcache.timeout.ms	How long to wait for a JMX response	long	500
lenses.interval.sql.udf	How often to look for new UDF/UDAF (user defined [aggregate] functions)	long	10000
lenses.kafka.consumers.batch.size	How many consumer groups to retrieve in a single request	Int	500
lenses.kafka.ws.heartbeat.ms	How often to send heartbeat messages in TCP connection	long	30000
lenses.kafka.ws.poll.ms	Max time for kafka consumer data polling on WS APIs	long	10000
lenses.kubernetes.config.reload.interval	Time interval to reload the Kubernetes configuration file.	long	30000
lenses.kubernetes.watch.reconnect.limit	How many times to reconnect to Kubernetes Watcher before considering the cluster unavailable	long	10
lenses.kubernetes.watch.reconnect.interval	How often to wait between Kubernetes Watcher reconnection attempts	long	5000
lenses.kubernetes.websocket.timeout	How long to wait for a Kubernetes Websocket response	long	15000
lenses.kubernetes.websocket.ping.interval	How often to ping Kubernetes Websocket to check it’s alive	long	30000
lenses.akka.request.timeout.ms	Max time for a response in an Akka Actor	long	10000
lenses.sql.monitor.frequency	How often to emit healthcheck and performance metrics on Streaming SQL	long	10000
lenses.audit.data.access	Record dataset access as audit log entries	boolean	true
lenses.audit.data.max.records	How many dataset view entries to retain in the audit log. Set to -1 to retain indefinitely	int	500000
lenses.explore.lucene.max.clause.count	Override Lucene’s maximum number of clauses permitted per BooleanQuery	int	1024
lenses.explore.queue.size	Optional setting to bound Lenses internal queue used by the catalog subsystem. It needs to be positive integer or it will be ignored.	int	N/A
lenses.interval.kafka.connect.http.timeout.ms	How long to wait for Kafka Connect response to be retrieved	int	10000
lenses.interval.kafka.connect.healthcheck	How often to check the Kafka health	int	15000
lenses.interval.schema.registry.http.timeout.ms	How long to wait for Schema Registry response to be retrieved	int	10000
lenses.interval.zookeeper.healthcheck	How often to check the Zookeeper health	int	15000
lenses.ui.topics.row.limit	The number of Kafka records to load automatically when exploring a topic	int	200
lenses.deployments.connect.failure.alert.check.interval	Time interval in seconds to check the connector failure grace period has completed. Used by the Connect auto-restart failed connectors functionality. It needs too be a value between (1,600].	int	10
lenses.provisioning.path	Folder on the filesystem containing the provisioning data. See [provisioning docs](link to provisioning docs) for further details	string
lenses.provisioning.interval	Time interval in seconds to check for changes on the provisioning resources	int
lenses.schema.registry.client.http.retryOnTooManyRequest	When enabled, Lenses will retry a request whenever the schema registry returns a 429 Too Many Requests	boolean	false
lenses.schema.registry.client.http.maxRetryAwait	Max amount of time to wait whenever a 429 Too Many Requests is returned.	duration	"2 seconds"
lenses.schema.registry.client.http.maxRetryCount	Max retry count whenever a 429 Too Many Requests is returned.	integer	2
lenses.schema.registry.client.http.rate.type	Specifies if http requests to the configured schema registry should be rate limited. Can be "session" or "unlimited"	"unlimited" | "session"	unlimited
lenses.schema.registry.client.http.rate.maxRequests	Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.	integer	N/A
lenses.schema.registry.client.http.rate.window	Whenever the rate limiter is "session" this configuration will determine the duration of the window used.	duration	N/A
lenses.schema.connect.client.http.retryOnTooManyRequest	Retry a request whenever a connect cluster returns a 429 Too Many Requests	boolean	false
lenses.schema.connect.client.http.maxRetryAwait	Max amount of time to wait whenever a 429 Too Many Requests is returned.	duration	2 seconds
lenses.schema.connect.client.http.maxRetryCount	Max retry count whenever a 429 Too Many Requests is returned.	integer	2
lenses.connect.client.http.rate.type	Specifies if http requests to the configured connect cluster should be rate limited. Can be "session" or "unlimited"	"unlimited" | "session"	unlimited
lenses.connect.client.http.rate.maxRequests	Whenever the rate limiter is "session" this configuration will determine the max amount of requests per window size that are allowed.	integer	N/A
lenses.connect.client.http.rate.window	Whenever the rate limiter is "session" this configuration will determine the duration of the window used.	duration	N/A

Connectors topology

Control how Lenses identifies your connectors in the Topology view. Catalogue your connector types, set their icons, and control how Lenses extracts the topics used by your connectors.

Lenses comes preconfigured for some of the popular connectors as well as the Stream Reactor connectors. If you see that Lenses doesn’t automatically identify your connector type then use the lenses.connectors.info setting to register it with Lenses.

Add a new HOCON object {} for every new Connector in your lenses.connectors.info list :

  lenses.connectors.info = [
      {
        class.name = "The connector full classpath"
        name = "The name which will be presented in the UI"
        instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
        sink = true
        extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
        icon = "file.png"
        description = "A description for the connector"
        author = "The connector author"
      }
  ]

This configuration allows the connector to work with the topology graph, and also have the RBAC rules applied to it.

Source example

To extract the topic information from the connector configuration, source connectors require an extra configuration. The extractor class should be: io.lenses.config.kafka.connect.SimpleTopicsExtractor. Using this extractor requires an extra property configuration. It specifies the field in the connector configuration which determines the topics data is sent to.

Here is an example for the file source:

  lenses.connectors.info = [
    {
      class.name = "org.apache.kafka.connect.file.FileStreamSource"
      name = "File"
      instance = "file"
      sink = false
      property = "topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
    }
  ]

Sink example

An example of a Splunk sink connector and a Debezium SQL server connector

  lenses.connectors.info = [
    {
      class.name = "com.splunk.kafka.connect.SplunkSinkConnector"
      name = "Splunk Sink",
      instance = "splunk.hec.uri"
      sink = true,
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "splunk.png",
      description = "Stores Kafka data in Splunk"
      docs = "https://github.com/splunk/kafka-connect-splunk",
      author = "Splunk"
    },
    {
      class.name = "io.debezium.connector.sqlserver.SqlServerConnector"
      name = "CDC MySQL"
      instance = "database.hostname"
      sink = false,
      property = "database.history.kafka.topic"
      extractor.class = "io.lenses.config.kafka.connect.SimpleTopicsExtractor"
      icon = "debezium.png"
      description = "CDC data from RDBMS into Kafka"
      docs = "//debezium.io/docs/connectors/mysql/",
      author = "Debezium"
    }
  ]

External Applications

Key	Description	Default	Type	Required
apps.external.http.state.refresh.ms	When registering a runner for external app, a health-check interval can be specified. If it is not, this default interval is used (value in milliseconds)	30000	int	no
apps.external.http.state.cache.expiration.ms	Last known state of the runner is stored in a cache. The entries in the cache are being invalidated after a time that is defined by following configuration key (value in milliseconds). This value should not be lower than the apps.external.http.state.refresh.ms value.	60000	int	no