Deploying an Agent

This page describes deploying an Lenses Agent via Docker.

The Agent docker image can be configured via environment variables or via volume mounts for the configuration files.

Running the Docker

Please check "File configuration" before running the command

terminal

docker run --name lenses-agent \
-v $(pwd)/provisioning.yaml:/mnt/provision-secrets/provisioning.yaml \
-v $(pwd)/lenses-agent.conf:/data/lenses-agent.conf \
-e LENSES_PROVISIONING_PATH=/mnt/provision-secrets \
lensesio/lenses-agent:6.0

---

Provisioning

1

Configure HQ

To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

Create new Environment and obtain agent key

TLS Enabled

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: [LENSES_HQ_URL]
      port:
        value: 10000
      agentKey:
        value: ${LENSES_HQ_AGENT_KEY}
      sslEnabled:
        value: true
      sslTruststore:
        file: "hq-truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

TLS Disabled

Agent key reference

Agent key within provisioning.yaml can be referenced as a:

• environment variable shown in example above

• inline string

2

Configure Kafka

Bare in mind that each Agent connects to one Kafka cluster.

There are many Kafka flavours today in the market. Good news is that Lenses support all flavours of Kafka and we are trying hard to keep documentation up to date.

In the following link you can find provisioning examples for the most common Kafka flavours.

Other connections

There are also provisioning examples for other components:

• Schema Registry

• Kafka Connect

• Zookeeper

• AWS

• Alerts & Audit integrations

3

Configure Database

Bare in mind that each Agent requires separate database. Two Agents sharing the same database can lead to racing condition issue.

Last step is to configure a database to which Agent will be connecting to.

Running Agent with Postgres database

lenses-agent.conf

lenses.storage.postgres.host="dbname"
lenses.storage.postgres.database="agentdb"
lenses.storage.postgres.username="agentusername"
lenses.storage.postgres.password="changeme" 
lenses.storage.postgres.port="5432"
lenses.storage.postgres.schema="myschema"

Running Agent with H2 embedded database

Embedded database is not recommended to be used in Production or high load environments.

In order to run Agent with embedded database, your lenses-agent.conf should not carry any property starting with lenses.storage.postgres.*

---

Provisioning example file

This provisioning file includes connections to:

• Kafka Cluster

• Schema Registry

• Kafka Connect

• Lenses HQ

provisioning.yaml

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: agent_key_*
    port:
      value: 10000
    server:
      value: current-lenses-hq.panoptes.svc.cluster.local
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://prod-1-kafka-bootstrap.kafka-prod.svc.cluster.local:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
    protocol:
      value: PLAINTEXT
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1
confluentSchemaRegistry:
- configuration:
    schemaRegistryUrls:
      value:
      - http://schema-registry-prod.kafka-prod.svc.cluster.local:8081
  name: schema-registry
  tags:
  - prod
  - global
  version: 1
connect:
- configuration:
    workers:
      value:
      - http://prod-1-connect-connect.kafka-prod.svc.cluster.local:8083
  name: datalake-connect
  tags:
  - prod
  - us
  version: 1

---

File configuration

In order for command above provisioning yaml has to be configured.

There are two mandatory connections:

1. HQ which requires AgentKey (LENSESHQ_AGENT_KEY) - this key is being created once user registers "New environment" in HQ;

2. Kafka connection

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: ${LENSESHQ_AGENT_KEY}
    port:
      value: 10000
    server:
      value: hq
  name: lenses-hq
  tags:
  - hq
  version: 1

kafka:
- configuration:
   kafkaBootstrapServers:
      value:
      - PLAINTEXT://kafka:9092
   protocol:
      value: PLAINTEXT
  name: kafka
  tags:
  - demo
  version: 1

---

Environment Variables

Environment variables prefixed with LENSES_ are transformed into corresponding configuration options. The environment variable name is converted to lowercase and underscores (_) are replaced with dots (.). As an example set the option lenses.port use the environment variable LENSES_PORT.

Alternatively, the lenses-agent.conf can be mounted directly as

• /mnt/settings/lenses-agent.conf

---

Docker volumes

The Docker image exposes four volumes in total, where cache, logs, plugins, and persistent data are stored:

• /data/storage

• /data/plugins

• /data/logs

• /data/kafka-streams-state

Storage volume

Resides under /data/storage and is used to store persistent data, such as Data Policies. For this data to survive between Docker runs and/or Agent upgrades, the volume must be managed externally (persistent volume).

Plugins volume

Resides under /data/plugins it’s where classes that extend Agent may be added —such as custom Serdes, LDAP filters, UDFs for the Lenses SQL table engine, and custom_http implementations.

Logs volume

Resides under /data/logs, logs are stored here. The application also logs to stdout, so the log files aren’t needed for most cases.

KStreams state volume

Resides under /data/kafka-streams-state, used when Lenses SQL is in IN_PROC configuration. In such a case, Lenses uses this scratch directory to cache Lenses SQL internal state. Whilst this directory can safely be removed, it can be beneficial to keep it around, so the Processors won’t have to rebuild their state during a restart.

---

Agent TLS and Global JVM Trust Store

By default, the the serves connections over plaintext (HTTP). It is possible to use TLS instead. The Docker image offers the ability to provide the content for extra files via secrets mounted as files or as environment variables. Especially for SSL, Docker supports SSL/TLS keys and certificates in Java Keystore (JKS) formats.

This capability is optional, and users can mount such files under custom paths and configure lenses-agent.conf manually via environment variables, or lenses.append.conf.

There are two ways to use the File/Variable names of the table below.

1. Create a file with the appropriate filename as listed below and mount it under /mnt/settings, /mnt/secrets, or /run/secrets

2. Set them as environment variables.

All settings except for passwords, can be optionally encoded in base64. The docker will detect such encoding automatically.

File / Variable Name	Description
FILECONTENT_JVM_SSL_TRUSTSTORE	The SSL/TLS trust store to use as the global JVM trust store. Add to LENSES_OPTS the property javax.net.ssl.trustStore
FILECONTENT_JVM_SSL_TRUSTSTORE_PASSWORD	Τhe trust store password. If set, the startup script will add automatically to LENSESOPTS the property javax.net.ssl.trustStorePassword (**_base64 not supported**)
FILECONTENT_LENSES_SSL_KEYSTORE	The SSL/TLS keystore to use for the TLS listener for the Agent

---

Process UID/GUI

The docker does not require running as root. The default user is set to root for convenience and to verify upon start-up that all the directories and files have the correct permissions. The user drops to nobody and group nogroup (65534:65534) before starting the Agent.

If the image is started without root privileges, the agent will start successfully using the effective uid:gid applied. Ensure any volumes mounted (i.e., for the settings and data) have the correct permission set.