Install

This page describes configuring and starting Lenses HQ and Agent against your Kafka cluster.

This guide uses the Lenses docker-compose file. For non-dev installations and automation see the Installation section.

Configure HQ

HQ is configured via by one file, config.yaml. The docker-compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

You only need to follow this step if you do not want to use the local Postgres instance started by the docker-compose file.

You must create a database and role in your Postgres instance for HQ to use. See Database Role.

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

docker-compose.yaml

 hq.config.yaml:
    content: |
      # ACCEPT THE LENSES EULA
      license:
        acceptEULA: true
      database:
        host: postgres:5432
        username: [YOUR_POSTGRES_YOUR_NAME]
        password: lenses
        database: hq

Authentication

Currently HQ supports:

1. Basic Authentication (default)

2. SAML

For this example, we will use basic authentication, for information on configuring other methods, see Authentication and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

terminal

docker-compose up hq

You can now log in to your browser with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

1. Login into HQ and create an environment, Environments->New Environment.

2. At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

terminal

➜  lenses environments
Manage Environments.

Usage:
  lenses environments [command]

Aliases:
  environments, e, env, envs

Available Commands:
  create      Creates a new environment.
  delete      Deletes an environment.
  get         Retrieves a single environment by name.
  list        Lists all environments
  metadata    Manages environment metadata.
  update      Updates an environment.
  watch       Watch live environment updates.

Configure the Agent

The Agent is configured via two files:

• lenses.conf - holds low-level configuration options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file

• provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

You only need to follow this step if you do not want to use the local Postgres instance started by the docker-compose file.

You must create a database and role in your Postgres instance for the Agent to use. See Database Role.

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

docker-compose.yaml

 agent.lenses.conf:
    content: |
      lenses.storage.postgres.host=[YOUR_POSTGRES_INSTANCE]
      lenses.storage.postgres.port=[YOUR_POSTGRES_PORT]
      lenses.storage.postgres.database=agent
      lenses.storage.postgres.username=lenses
      lenses.storage.postgres.password=lenses

Connect the Agent to HQ

You can connect the agent to HQ in two ways, all via provisioning

1. Start the Agent docker with the an AGENT_KEY via environment variables at minimum. You need to create an environment in HQ to get this key

2. Or mount a provisioning file that contains the connection to HQ, recommended for TLS enabled HQs

You can still reference environment variables if you mount the file, e.g

agentKey:
  value: ${LENSES_HQ_AGENT_KEY

Minimal Start

First deploy HQ and create an environment, then with the AGENT KEY run:

docker run \                                                                          
  --name "xxx" \
  --network=lenses \
  --restart=unless-stopped \
  -e PROVISION_AGENT_KEY=YOUR_AGENT_KEY \
  -e PROVISION_HQ_URL=YOUR_LENSES_HQ_URL \
  lensesio/lenses-agent:latest 

This will start and connect to HQ but not to Kafka or other services. It will create a provisioning file in data/provisioning.

Set the docker network accordingly

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this, update the provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

You can add connections in three ways:

1. direct editing of the provisioning file directly

2. Lenses UX

3. APIs (which step 2 uses)

They all result in writing a provisioning file which the Agent picks up and loads.

Manual file editing

You must manual add all the connections you want to the file and then mount it. To help you create a provisioning file you can use the JSON SCHEMA support. In you IDE, like VS Code, create a file called provisioning.yaml and add the following line at the top:

# yaml-language-server: $schema=./agent/provisioning.schema-6.1.json

Then start typing, for example k for Kafka and Kafka Connect, s for schema registry or just crtl+space to trigger the default templates.

Fill in the required fields, your editor should highlight issues for you

Start with provisioning

Add lenses-agent.conf if you are overriding defaults like the embedded database.

terminal

docker run --name lenses-agent \
-v $(pwd)/provisioning.yaml:/mnt/provision-secrets/provisioning.yaml \
-v $(pwd)/lenses-agent.conf:/data/lenses-agent.conf \
-e LENSES_PROVISIONING_PATH=/mnt/provision-secrets \
lensesio/lenses-agent:latest

See provisioning for examples of different authentication types for Kafka.

Lenses UX

When you create an environment via Lenses UI, you will be guided through the process to, start the agent, and configuration the connections. The experience is similar to manually editing the provisioning but it uses the APIs to push down and test configurations.

APIs

You can also use the APIs directly. See here.