Deploying an Agent

To install the Agent from the archive you must:

1. Extract the archive

2. Configure the Agent

3. Start the Agent

Extracting the archive

Extract the archive using the following command

tar -xvf lenses.tar.gz -C lenses

Inside the extract archive, you will find.

   lenses
   ├── lenses.conf       ← edited and renamed from .sample
   ├── logback.xml
   ├── logback-debug.xml
   ├── bin/
   ├── lib/
   ├── licences/
   ├── logs/             ← created when you run Lenses
   ├── plugins/
   ├── storage/          ← created when you run Lenses
   └── ui/

Configure the Agent

To configure the agents connection to Postgres and its provisioning file. See here in the quickstart.

Once the agent files are configure you can continue to start the agent.

Provisioning

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: [LENSES_HQ_URL]
      port:
        value: 10000
      agentKey:
        value: ${LENSESHQ_AGENT_KEY}
      sslEnabled:
        value: true
      sslTruststore:
        file: "/mnt/provision-secrets/hq/truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

Starting the Agent

Start Lenses by running:

bin/lenses

or pass the location of the config file:

bin/lenses lenses.conf

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir

If you do not pass the location of lenses.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the lenses.conf to be readable only by the lenses user.

chmod 0600 /path/to/lenses.conf
chown [lenses-user]:root /path/to/lenses.conf

The agent needs write access in 4-5 places in total:

1. [RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:

   1. [RUNTIME DIRECTORY]/logs Where logs are stored

   2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.

   3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.

   4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.

   5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

1. Mount /tmp without noexec

2. or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

LENSES_OPTS="-Dorg.xerial.snappy.tempdir=/path/to/exec/tmp -Djava.io.tmpdir=/path/to/exec/tmp"

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

[Unit]
Description=Run Lenses.io service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback.xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

export LENSES_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
bin/lenses

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 8GB RAM /4 CPUs and 20GB disk space.