1 of 3

Linux

This page describes install the Lenses via a Linux archive

Deploying HQ

This page describes the install of the Lenses Agent via an archive on Linux.

To install the HQ from the archive you must:

Extract the archive
Configure the HQ
Start the HQ

Extracting the archive

Extract the archive using the following command

terminal

tar -xvf hq-v6.tar.gz -C lenses-hq

Inside the extract archive, you will find.

terminal

   lenses-hq
   ├── backend-darwin-arm64

Configuring the HQ

In order to properly configure HQ, two core components are necessary:

Postgres database

Configure Authentication

To set up authentication, there are multiple methods available.

You can choose between:

password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.

Both password based and SAML / SSO authentication methods can be used alongside each other.

First to cover is users property. Users Property: The users property is defined as an array, where each entry includes a username and a password. The passwords are hashed using bcrypt for security purposes, ensuring that they are stored securely.

Second to cover will be administrators. It serves as definition of user emails which will have highest level of permissions upon authentication to HQ.

config.yaml

auth:
  users:
    - username: admin
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple").
  administrators:
    - admin
    - admin@example.com
  saml:
    enabled: true
    metadata: |-
      <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor>
      ...
      ...
      </md:EntityDescriptor>
    baseURL: https://example.com
    entityID: https://example.com
    userCreationMode: sso
    groupMembershipMode: sso

Configure HTTP endpoint

Another part which has to be set in order to successfully run HQ is the http definition. As previously mentioned, this parameter defines everything around HTTP endpoint of the HQ itself and how users will interact with.

Definition of HTTP object is as follows:

config.yaml

http:
  address: :8080
  accessControlAllowOrigin:
    - https://example.com
  accessControlAllowCredentials: false
  secureSessionCookies: true
  tls:
    enabled: true
    cert: ""
    key: ""

More about setting up TLS can be read here.

Configure Agent endpoint

After correctly configuring authentication strategy and connection endpoint , agent handling is the last most important box to tick.

The Agent's object is defined as follows:

config.yaml

agents:
  address: :10000
  tls:
    enabled: true
    cert: ""
    key: ""

More about setting up TLS can be read here.

Configure database

Prerequisite:

Running Postgres instance;
Created database for HQ;
Username (and password) which has access to created database;

In order to successfully run HQ, storage within config.yaml has to be defined first.

Definition of storage object is as follows:

config.yaml

database:
  host: postgres:5432
  username: panoptes
  password: password
  database: panoptes
  params:
    sslmode: require

Configure license and accept EULA

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

config.yaml

license:
  key: license_key_*
  acceptEULA: true

Final Configuration File

If you have meticulously followed all the outlined steps, your config.yaml file should mirror the example provided below, fully configured and ready for deployment. This ensures your system is set up correctly with all necessary settings for authentication, database connection, and other configurations optimally defined.

config.yaml

auth:
  users:
    - username: admin
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple").
  administrators:
    - admin
    - admin@example.com
  saml:
    enabled: true
    metadata: |-
      <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor>
      ...
      ...
      </md:EntityDescriptor>
    baseURL: https://example.com
    entityID: https://example.com
    userCreationMode: sso
    groupMembershipMode: sso
http:
  address: :8080
  accessControlAllowOrigin:
    - https://example.com
agents:
  address: :10000
database:
  host: postgres:5432
  username: panoptes
  password: password
  database: panoptes
  params:
    sslmode: require
licence:
  key: license_key_*
logger:
  mode: text
  level: debug

Starting the HQ

Start Lenses by running:

terminal

./backend-darwin-amd64

or pass the location of the config file:

terminal

./backend-darwin-amd64 config.yaml

If you do not pass the location of the config file, the HQ will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

Once HQ starts, it will be listening on the https://localhost:8080

To stop HQ, press CTRL+C.

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

[Unit]
Description=Run Lenses.io service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses-hq
ExecStart=/opt/lenses-hq /etc/lenses-hq/config.yaml

[Install]
WantedBy=multi-user.target

Deploying an Agent

This page describes the install of the Lenses Agent via an archive on Linux.

To install the Agent from the archive you must:

Extract the archive
Configure the Agent
Start the Agent

Extracting the archive

Extract the archive using the following command

Inside the extract archive, you will find.

Configure the Agent

Once the agent files are configure you can continue to start the agent.

The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.

Provisioning

Configure HQ

To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

Agent key reference

Agent key within provisioning.yaml can be referenced as a:

environment variable shown in example above
inline string

Configure Kafka

There are many Kafka flavours today in the market. Good news is that Lenses support all flavours of Kafka and we are trying hard to keep documention up to date.

Other connections

There are also provisioning examples for other components:

Starting the Agent

Start Lenses by running:

or pass the location of the config file:

Provisioning file path

If you configured provisioning.yaml make sure to place following property:

If you do not pass the location of lenses.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the lenses.conf to be readable only by the lenses user.

The agent needs write access in 4-5 places in total:

[RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:
1. [RUNTIME DIRECTORY]/logs Where logs are stored
2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.
3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.
4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.
5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

Back-up this location for disaster recovery

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

Mount /tmp without noexec
or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

Global Truststore

The Agent uses the default trust store (cacerts) of the system’s JRE (Java Runtime) installation. The trust store is used to verify remote servers on TLS connections, such as Kafka Brokers with an SSL protocol, JMX over TLS, and more. Whilst for some types of connections (e.g. Kafka Brokers) a separate keystore can be provided at the connection’s configuration, for some other connections (JMX over TLS) we always rely on the system trust store.

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

Increase as a super-user the soft limit to 4096 with:

Use 8GB RAM /4 CPUs and 20GB disk space.

Deploying HQ

This page describes the install of the Lenses Agent via an archive on Linux.

To install the HQ from the archive you must:

Extract the archive
Configure the HQ
Start the HQ

Extracting the archive

Extract the archive using the following command

terminal

tar -xvf hq-v6.tar.gz -C lenses-hq

Inside the extract archive, you will find.

terminal

   lenses-hq
   ├── backend-darwin-arm64

Configuring the HQ

In order to properly configure HQ, two core components are necessary:

Postgres database

Configure Authentication

To set up authentication, there are multiple methods available.

You can choose between:

password-based authentication, which requires users to provide a username and password;
and SAML/SSO (Single Sign-On) authentication, which allows users to authenticate through an external identity provider for a seamless and secure login experience.

Both password based and SAML / SSO authentication methods can be used alongside each other.

Second to cover will be administrators. It serves as definition of user emails which will have highest level of permissions upon authentication to HQ.

config.yaml

auth:
  users:
    - username: admin
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple").
  administrators:
    - admin
    - admin@example.com
  saml:
    enabled: true
    metadata: |-
      <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor>
      ...
      ...
      </md:EntityDescriptor>
    baseURL: https://example.com
    entityID: https://example.com
    userCreationMode: sso
    groupMembershipMode: sso

Configure HTTP endpoint

Definition of HTTP object is as follows:

config.yaml

http:
  address: :8080
  accessControlAllowOrigin:
    - https://example.com
  accessControlAllowCredentials: false
  secureSessionCookies: true
  tls:
    enabled: true
    cert: ""
    key: ""

More about setting up TLS can be read here.

Configure Agent endpoint

After correctly configuring authentication strategy and connection endpoint , agent handling is the last most important box to tick.

The Agent's object is defined as follows:

config.yaml

agents:
  address: :10000
  tls:
    enabled: true
    cert: ""
    key: ""

More about setting up TLS can be read here.

Configure database

Prerequisite:

Running Postgres instance;
Created database for HQ;
Username (and password) which has access to created database;

In order to successfully run HQ, storage within config.yaml has to be defined first.

Definition of storage object is as follows:

config.yaml

database:
  host: postgres:5432
  username: panoptes
  password: password
  database: panoptes
  params:
    sslmode: require

Configure license and accept EULA

In demo purposes and testing the product you can use our community license

license_key_2SFZ0BesCNu6NFv0-EOSIvY22ChSzNWXa5nSds2l4z3y7aBgRPKCVnaeMlS57hHNVboR2kKaQ8Mtv1LFt0MPBBACGhDT5If8PmTraUM5xXLz4MYv

config.yaml

license:
  key: license_key_*
  acceptEULA: true

Final Configuration File

config.yaml

auth:
  users:
    - username: admin
      password: $2a$10$F66cb6ZhnJjGCZuxlvKP1e84eytTpT1MDJcpBblHaZgsqp1/Aa0LG # bcrypt("correcthorsebatterystaple").
  administrators:
    - admin
    - admin@example.com
  saml:
    enabled: true
    metadata: |-
      <?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor>
      ...
      ...
      </md:EntityDescriptor>
    baseURL: https://example.com
    entityID: https://example.com
    userCreationMode: sso
    groupMembershipMode: sso
http:
  address: :8080
  accessControlAllowOrigin:
    - https://example.com
agents:
  address: :10000
database:
  host: postgres:5432
  username: panoptes
  password: password
  database: panoptes
  params:
    sslmode: require
licence:
  key: license_key_*
logger:
  mode: text
  level: debug

Starting the HQ

Start Lenses by running:

terminal

./backend-darwin-amd64

or pass the location of the config file:

terminal

./backend-darwin-amd64 config.yaml

If you do not pass the location of the config file, the HQ will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

Once HQ starts, it will be listening on the https://localhost:8080

To stop HQ, press CTRL+C.

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

[Unit]
Description=Run Lenses.io service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses-hq
ExecStart=/opt/lenses-hq /etc/lenses-hq/config.yaml

[Install]
WantedBy=multi-user.target

Deploying an Agent

This page describes the install of the Lenses Agent via an archive on Linux.

To install the Agent from the archive you must:

Extract the archive
Configure the Agent
Start the Agent

Extracting the archive

Extract the archive using the following command

Inside the extract archive, you will find.

Configure the Agent

To configure the agents connection to Postgres and its provisioning file. See here in the .

Once the agent files are configure you can continue to start the agent.

The configuration files are the same for docker and Linux, for docker we are simply mounting the files into the container.

Provisioning

Configure HQ

To see be able to view and drilling to your Kafka environment, you need to connect the agent to HQ. You need to create an environment in HQ and copy the Agent Key into the provisioning.yaml.

provisioning.yaml

lensesHq:
  - name: lenses-hq
    version: 1
    tags: ['hq']
    configuration:
      server:
        value: [LENSES_HQ_URL]
      port:
        value: 10000
      agentKey:
        value: ${LENSESHQ_AGENT_KEY}
      sslEnabled:
        value: true
      sslTruststore:
        file: "/mnt/provision-secrets/hq/truststore.jks"
      sslTruststorePassword:
        value: ${LENSES_HQ_AGENT_TRUSTSTORE_PWD}

Agent key reference

Agent key within provisioning.yaml can be referenced as a:

environment variable shown in example above
inline string

Configure Kafka

There are many Kafka flavours today in the market. Good news is that Lenses support all flavours of Kafka and we are trying hard to keep documention up to date.

In the following you can find provisioning examples for the most common Kafka flavours.

Other connections

There are also provisioning examples for other components:

Starting the Agent

Start Lenses by running:

terminal

bin/lenses

or pass the location of the config file:

terminal

bin/lenses lenses.conf

Provisioning file path

If you configured provisioning.yaml make sure to place following property:

lenses.conf

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir

If you do not pass the location of lenses.conf, the Agent will look for it inside the current (runtime) directory. If it does not exist, it will try its installation directory.

To stop Lenses, press CTRL+C.

File permissions

Set the permissions of the lenses.conf to be readable only by the lenses user.

chmod 0600 /path/to/lenses.conf
chown [lenses-user]:root /path/to/lenses.conf

The agent needs write access in 4-5 places in total:

[RUNTIME DIRECTORY] When the Agent runs, it will create at least one directory under the directory it is run in:
1. [RUNTIME DIRECTORY]/logs Where logs are stored
2. [RUNTIME DIRECTORY]/logs/sql-kstream-state Where SQL processors (when In Process mode) store state. To change the location for the processors’ state directory, use lenses.sql.state.dir option.
3. [RUNTIME DIRECTORY]/storage Where the H2 embedded database is stored when PostgreSQL is not set. To change this directory, use the lenses.storage.directory option.
4. /run (Global directory for temporary data at runtime) Used for temporary files. If Lenses does not have permission to use it, it will fall back to /tmp.
5. /tmp (Global temporary directory) Used for temporary files (if access /run fails), and JNI shared libraries.

Back-up this location for disaster recovery

JNI libraries

The Agent and Kafka use two common Java libraries that take advantage of JNI and are extracted to /tmp.

You must either:

Mount /tmp without noexec
or set org.xerial.snappy.tempdir and java.io.tmpdir to a different location

LENSES_OPTS="-Dorg.xerial.snappy.tempdir=/path/to/exec/tmp -Djava.io.tmpdir=/path/to/exec/tmp"

SystemD example

If your server uses systemd as a Service Manager, then manage the Agent (start upon system boot, stop, restart). Below is a simple unit file that starts the Agent automatically on system boot.

[Unit]
Description=Run Lenses.io service

[Service]
Restart=always
User=[LENSES-USER]
Group=[LENSES-GROUP]
LimitNOFILE=4096
WorkingDirectory=/opt/lenses
#Environment=LENSES_LOG4J_OPTS="-Dlogback.configurationFile=file:/etc/lenses/logback.xml"
ExecStart=/opt/lenses/bin/lenses /etc/lenses/lenses.conf

[Install]
WantedBy=multi-user.target

Global Truststore

It is possible to set up a global custom trust store via the LENSES_OPTS environment variable:

export LENSES_OPTS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
bin/lenses

Hardware & OS

Run on any Linux server (review ulimits or container technology (docker/kubernetes). For RHEL 6.x and CentOS 6.x use docker.

Linux machines typically have a soft limit of 1024 open file descriptors. Check your current limit with the ulimit command:

ulimit -S -n     # soft limit
ulimit -H -n     # hard limit

Increase as a super-user the soft limit to 4096 with:

ulimit -S -n 4096

Use 8GB RAM /4 CPUs and 20GB disk space.