Install

This page describes configuring and starting Lenses HQ and Agent against your Kafka cluster.

This guide is using the Lenses docker compose file. For non dev installations and automation see the Installation section.

Configure HQ

HQ is configured via by one file, config.yaml. The docker compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

You only need to follow this step if you do not want to use the local postgres instance started by the docker compose file.

You must create a database and role in your postgres instance for HQ to use. See Database Role.

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

docker-compose.yaml

 hq.config.yaml:
    content: |
      # ACCEPT THE LENSES EULA
      license:
        acceptEULA: true
      database:
        host: postgres:5432
        username: [YOUR_POSTGRES_YOUR_NAME]
        password: lenses
        database: hq

Authentication

Currently HQ supports:

1. Basic Authentication (default)

2. SAML

For this example we will use basic authentication, for information on configuring other methods, see Authentication and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

terminal

docker-compose up hq

You can now log in your browser with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

1. Login into HQ and create an environment, Environments->New Environment.

2. At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

terminal

➜  out ./hq-linux-amd64 environments
Manage Environments.

Usage:
  hq environments [command]

Aliases:
  environments, e, env, envs

Available Commands:
  create      Creates a new environment.
  delete      Deletes an environment.
  get         Retrieves a single environment by name.
  list        Lists all environments
  metadata    Manages environment metadata.
  update      Updates an environment.
  watch       Watch live environment updates.

Configure the Agent

The Agent is configured via two files:

• lenses.conf - holds low-level configuration options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file

• provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

You only need to follow this step if you do not want to use the local postgres instance started by the docker compose file.

You must create a database and role in your postgres instance for the Agent to use. See Database Role.

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

docker-compose.yaml

 agent.lenses.conf:
    content: |
      lenses.storage.postgres.host=[YOUR_POSTGRES_INSTANCE]
      lenses.storage.postgres.port=[YOUR_POSTGRES_PORT]
      lenses.storage.postgres.database=agent
      lenses.storage.postgres.username=lenses
      lenses.storage.postgres.password=lenses

Adding a HQ Connection

The Agent Key for an environment needs to be added to the agent.provisioning.yaml key in the docker compose file.

docker-compose.yaml

agent.provisioning.yaml:
    content: |
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace ${{LENSESHQ_AGENT_KEY}} with the Agent Key for the environment that you want to link to.

For more information on the configuration of the connection to HQ see here.

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this update the agent.provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

See provisioning for examples of different authentication types for Kafka.

1. Add the following for a basic plaintext connection to a Kafka broker, if you are using a different authentication mechanism adjust accordingly.

2. Remove, or adjust the Kafka (kafka-demo), Schema Registry and Connect services in the default docker-compose file.

docker-compose.yaml

agent.provisioning.yaml:
    content: |
      kafka:
        - name: kafka
          version: 1
          tags: [ 'kafka', 'dev' ]
          configuration:
            metricsType:
              value: JMX
            metricsPort:
              value: 9581
            kafkaBootstrapServers:
              value: PLAINTEXT://[YOUR_BOOTSTRAP_BROKER:PORT] 
            protocol:
              value: PLAINTEXT
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace [YOUR_BOOTSTRAP_BROKER:PORT] with the bootstrap brokers and ports for the Kafka cluster you want the Agent to connect to.

For examples of adding in other services such as Schema Registries and Kafka Connect see provisioning.

Start the Agent

To start Agent, run the following docker command:

For none dev environments, install the agent as close as possible to your Kafka clusters and automate the installation.

terminal

docker-compose up agent

Once the agent fully starts, it will report as connected in HQ, allowing you to explore your Kafka environments.