Deploying an Agent

This page describes the install of the Lenses Agent via an archive on Linux.

To install the Agent from the archive you must:

1. Extract the archive
2. Configure the Agent
3. Start the Agent

---

Extracting the archive

Installation link

Link to archives can be found here: https://archive.lenses.io/lenses/6.2/agent/

Extract the archive using the following command. It unpacks into a lenses-agent/ directory:

terminal

tar -xzf lenses-agent-latest-linux64.tar.gz

Inside the extracted archive, you will find.

   lenses-agent
   ├── lenses-agent.conf.sample   ← copy to lenses-agent.conf and edit
   ├── security.conf.sample       ← empty file (see "Starting the Agent")
   ├── logback.xml
   ├── logback-debug.xml
   ├── bin/              ← bin/lenses launcher
   ├── lib/
   ├── licenses/
   ├── logs/             ← created when you run Lenses
   ├── plugins/          ← create it yourself to add custom plugins to the classpath
   ├── storage/          ← created when you run Lenses (embedded DB)
   └── ui/

---

Configure the Agent

To configure the agent connection to Postgres and its provisioning file, see the quick start.

Once the agent files are configure you can continue to start the agent.

The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.

Data policies

For a new installation, configure the Agent to apply the data policies defined on IAM roles in Lenses HQ:

lenses.conf

lenses.data.policies.legacy.enabled = false

With this setting, the Agent masks data only for users whose groups have IAM roles with data policy statements. Create those roles in Lenses HQ before users read sensitive topics. See Data policies in IAM.

If you upgrade an existing Agent, keep the default true until you recreate its data policies on IAM roles. See Configuration Reference.

Connect the Agent to HQ

You can connect the agent to HQ in two ways, all via Provisioning

1. Start the Agent with the an AGENT_KEY via environment variables at minimum. You need to create an environment in HQ to get this key
2. Or provisioning file that contains the connection to HQ, recommended for TLS enabled HQs

You can still reference environment variables if you use the file, e.g

agentKey:
  value: ${LENSES_HQ_AGENT_KEY}

Database

By default the Agent will start with an embedded database. If you want to use Postgres, which is recommended for production, see Agent Database. Database settings are set in lenses-agent.conf.

---

Starting the Agent

Provisioning file path

If you configured provisioning.yaml make sure to place following property:

lenses.conf

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir

Start Lenses by running:

terminal

bin/lenses

or pass the location of the config file:

terminal

bin/lenses lenses-agent.conf

If you do not pass the location of lenses-agent.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

security.conf is required. The Agent will not start without it and fails with Security configuration path ... 'security.conf' can not be found. Its default path is relative to the process working directory, so create an empty file there (with systemd, that is the unit's WorkingDirectory; run interactively, it is the current directory):

touch security.conf

The file is expected to be empty from Lenses 6.0 onwards. Alternatively set an absolute path via lenses.secret.file.

To stop Lenses, press CTRL+C.

---

File permissions

Set the permissions of the lenses-agent.conf to be readable only by the lenses user.

chmod 0600 /path/to/lenses-agent.conf
chown [lenses-user]:root /path/to/lenses-agent.conf

The agent needs write access in 4-5 places in total:

1. [RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:

   1. [RUNTIME DIRECTORY]/logs Where logs are stored
   2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.
   3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.
   4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.
   5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

Back-up this location for disaster recovery

---

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

1. Mount /tmp without noexec
2. or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

LENSES_OPTS="-Dorg.xerial.snappy.tempdir=/path/to/exec/tmp -Djava.io.tmpdir=/path/to/exec/tmp"

---

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot. It assumes you moved the extracted lenses-agent directory to /opt/lenses-agent.

Older installations commonly used /opt/lenses instead. Either works — just keep WorkingDirectory and ExecStart pointing at the same directory.

[Unit]
Description=Run Agent service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses-agent
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback.xml"
ExecStart=/opt/lenses-agent/bin/lenses /etc/lenses/lenses-agent.conf

[Install]
WantedBy=multi-user.target

---

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

export LENSES_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
bin/lenses

---

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 8GB RAM /4 CPUs and 20GB disk space.