Deploying an Agent

This page describes the install of the Lenses Agent via an archive on Linux.

To install the Agent from the archive you must:

1. Extract the archive

2. Configure the Agent

3. Start the Agent

---

Extracting the archive

Installation link

Link to archives can be found here: https://archive.lenses.io/lenses/6.0/agent/

Extract the archive using the following command. It unpacks into a lenses-agent/ directory:

terminal

tar -xzf lenses-agent-latest-linux64.tar.gz

Inside the extracted archive, you will find.

   lenses-agent
   ├── lenses-agent.conf.sample   ← copy to lenses-agent.conf and edit
   ├── security.conf.sample       ← empty file (see "Starting the Agent")
   ├── logback.xml
   ├── logback-debug.xml
   ├── bin/              ← bin/lenses launcher
   ├── lib/
   ├── licenses/
   ├── logs/             ← created when you run Lenses
   ├── plugins/          ← create it yourself to add custom plugins to the classpath
   ├── storage/          ← created when you run Lenses (embedded DB)
   └── ui/

---

Configure the Agent

To configure the agents connection to Postgres and its provisioning file. See here in the quickstart.

Once the agent files are configure you can continue to start the agent.

The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.

Provisioning

1

Configure HQ

To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

Create new environment and obtain Agent key

TLS Enabled

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: [LENSES_HQ_URL]
      port:
        value: 10000
      agentKey:
        value: [LENSES_AGENT_KEY]
      sslEnabled:
        value: true
      sslTruststore:
        file: "hq-truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

TLS Disabled

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: [LENSES_HQ_URL]
      port:
        value: 10000
      agentKey:
        value: ${LENSES_HQ_AGENT_KEY}
      sslEnabled:
        value: false

Agent key reference

Agent key within provisioning.yaml can be referenced as a:

• environment variable shown in example above

• inline string

2

Configure Kafka Connection

Bare in mind that each Agent connects to one Kafka cluster.

There are many Kafka flavours today in the market. Good news is that Lenses support all flavours of Kafka and we are trying hard to keep documention up to date.

In the following link you can find provisioning examples for the most common Kafka flavours.

Other connections

There are also provisioning examples for other components:

• Schema Registry

• Kafka Connect

• Zookeeper

• AWS

• Alerts & Audit integrations

3

Configure Database

Bare in mind that each Agent requires separate database. Two Agents sharing the same database can lead to racing condition issue.

Last step is to configure a database to which Agent will be connecting to.

Running Agent with Postgres database

lenses-agent.conf

lenses.storage.postgres.host="dbname"
lenses.storage.postgres.database="agentdb"
lenses.storage.postgres.username="agentusername"
lenses.storage.postgres.password="changeme" 
lenses.storage.postgres.port="5432"
lenses.storage.postgres.schema="myschema"

Running Agent with H2 embedded database

Embedded database is not recommended to be used in Production or high load environments.

In order to run Agent with embedded database, your lenses-agent.conf should not carry any property starting with lenses.storage.postgres.*

---

Provisioning example file

This provisioning file includes connections to:

• Kafka Cluster

• Schema Registry

• Kafka Connect

• Lenses HQ

provisioning.yaml

provisioning.yaml

lensesHq:
- configuration:
    agentKey:
      value: agent_key_*
    port:
      value: 10000
    server:
      value: current-lenses-hq.panoptes.svc.cluster.local
    sslEnabled:
      value: false
  name: lenses-hq
  tags:
  - hq
  version: 1
kafka:
- configuration:
    kafkaBootstrapServers:
      value:
      - PLAINTEXT://prod-1-kafka-bootstrap.kafka-prod.svc.cluster.local:9092
    metricsPort:
      value: 9999
    metricsType:
      value: JMX
    protocol:
      value: PLAINTEXT
  name: kafka
  tags:
  - prod
  - prod-1
  - us
  version: 1
confluentSchemaRegistry:
- configuration:
    schemaRegistryUrls:
      value:
      - http://schema-registry-prod.kafka-prod.svc.cluster.local:8081
  name: schema-registry
  tags:
  - prod
  - global
  version: 1
connect:
- configuration:
    workers:
      value:
      - http://prod-1-connect-connect.kafka-prod.svc.cluster.local:8083
  name: datalake-connect
  tags:
  - prod
  - us
  version: 1

---

Starting the Agent

Provisioning file path

If you configured provisioning.yaml make sure to place following property:

lenses.conf

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir

Start Lenses by running:

terminal

bin/lenses

or pass the location of the config file:

terminal

bin/lenses lenses-agent.conf

If you do not pass the location of lenses-agent.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

security.conf is required. The Agent will not start without it and fails with Security configuration path ... 'security.conf' can not be found. Its default path is relative to the process working directory, so create an empty file there (with systemd, that is the unit's WorkingDirectory; run interactively, it is the current directory):

touch security.conf

The file is expected to be empty from Lenses 6.0 onwards. Alternatively set an absolute path via lenses.secret.file.

To stop Lenses, press CTRL+C.

---

File permissions

Set the permissions of the lenses-agent.conf to be readable only by the lenses user.

chmod 0600 /path/to/lenses-agent.conf
chown [lenses-user]:root /path/to/lenses-agent.conf

The agent needs write access in 4-5 places in total:

1. [RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:

   1. [RUNTIME DIRECTORY]/logs Where logs are stored

   2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.

   3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.

   4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.

   5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

Back-up this location for disaster recovery

---

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

1. Mount /tmp without noexec

2. or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

LENSES_OPTS="-Dorg.xerial.snappy.tempdir=/path/to/exec/tmp -Djava.io.tmpdir=/path/to/exec/tmp"

---

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot. It assumes you moved the extracted lenses-agent directory to /opt/lenses-agent.

Older installations commonly used /opt/lenses instead. Either works — just keep WorkingDirectory and ExecStart pointing at the same directory.

[Unit]
Description=Run Agent service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses-agent
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback.xml"
ExecStart=/opt/lenses-agent/bin/lenses /etc/lenses/lenses-agent.conf

[Install]
WantedBy=multi-user.target

---

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

export LENSES_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
bin/lenses

---

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 8GB RAM /4 CPUs and 20GB disk space.