The quick start is for local development, with a local Kafka. This guide takes you through manually deploying HQ and an Agent to connect to your Kafka clusters.

For more detailed guides on the Helm, Docker and Linux see here.

How to connect to your Kafka?

To deploy Lenses against your environments you need to:

Prerequisites

EULA acceptance

To start HQ and an Agent you have to accept the Lenses EULA.

For HQ, in the config.yaml set:

license:
  acceptEULA: true

Kafka

Any version of Apache Kafka (2.0 or newer) on-premise and on-cloud. Supported providers include:

1. Confluent Platform & Cloud

2. AWS MSK & AWS MSK Serverless

3. Aiven

4. IBM Event Streams

5. Azure HDInsight & EventHubs

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

Postgres

HQ and Agents can share the same instance, by either using a separate database or schema for HQ and each agent, depending on your networking needs.

1. Postgres server running version 9.6 or higher.

Database Role

The recommended configuration is to create a dedicated login role and database for the HQ and each Agent, setting the HQ or Agent role as the database or schema owner. Both the agent and HQ need credentials, create a role for each.

# login as superuser and add Lenses role and database
psql -U postgres -d postgres <<EOF
CREATE ROLE lenses_agent WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_agent OWNER lenses_agent;

CREATE ROLE lenses_hq WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_hq OWNER lenses_hq;
EOF

Networking

1. Web sockets - You may need to adjust your load balancer to allow them. See here.

2. JMX connectivity - Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are also supported, as well as JOLOKIA and Open Metrics (MSK).

For more enable JMX for Agent itself see here.

Kafka ACLs

The agent requires access to your Kafka cluster. If ACLs are enable you will need to allow the Agent access.

SSO (optional)

If you want to use SSO / SAML for authentication you will need the metadata.xml file from your provider. See Authentication for more information.

Configure HQ

HQ is configured via by one file, config.yaml. The docker compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

 hq.config.yaml:
    content: |
      # ACCEPT THE LENSES EULA
      license:
        acceptEULA: true
      database:
        host: postgres:5432
        username: [YOUR_POSTGRES_YOUR_NAME]
        password: lenses
        database: hq

Authentication

Currently HQ supports:

1. Basic Authentication (default)

2. SAML

For this example we will use basic authentication, for information on configuring other methods, see Authentication and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

docker-compose up hq

You can now log in your browser with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

1. Login into HQ and create an environment, Environments->New Environment.

2. At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

➜  out ./hq-linux-amd64 environments
Manage Environments.

Usage:
  hq environments [command]

Aliases:
  environments, e, env, envs

Available Commands:
  create      Creates a new environment.
  delete      Deletes an environment.
  get         Retrieves a single environment by name.
  list        Lists all environments
  metadata    Manages environment metadata.
  update      Updates an environment.
  watch       Watch live environment updates.

Configure the Agent

The Agent is configured via two files:

• lenses.conf - holds low-level configuration options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file

• provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

 agent.lenses.conf:
    content: |
      lenses.storage.postgres.host=[YOUR_POSTGRES_INSTANCE]
      lenses.storage.postgres.port=[YOUR_POSTGRES_PORT]
      lenses.storage.postgres.database=agent
      lenses.storage.postgres.username=lenses
      lenses.storage.postgres.password=lenses

Adding a HQ Connection

The Agent Key for an environment needs to be added to the agent.provisioning.yaml key in the docker compose file.

agent.provisioning.yaml:
    content: |
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace ${{LENSESHQ_AGENT_KEY}} with the Agent Key for the environment that you want to link to.

For more information on the configuration of the connection to HQ see here.

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this update the agent.provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

1. Add the following for a basic plaintext connection to a Kafka broker, if you are using a different authentication mechanism adjust accordingly.

2. Remove, or adjust the Kafka (kafka-demo), Schema Registry and Connect services in the default docker-compose file.

agent.provisioning.yaml:
    content: |
      kafka:
        - name: kafka
          version: 1
          tags: [ 'kafka', 'dev' ]
          configuration:
            metricsType:
              value: JMX
            metricsPort:
              value: 9581
            kafkaBootstrapServers:
              value: PLAINTEXT://[YOUR_BOOTSTRAP_BROKER:PORT] 
            protocol:
              value: PLAINTEXT
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace [YOUR_BOOTSTRAP_BROKER:PORT] with the bootstrap brokers and ports for the Kafka cluster you want the Agent to connect to.

Start the Agent

To start Agent, run the following docker command:

docker-compose up agent

Once the agent fully starts, it will report as connected in HQ, allowing you to explore your Kafka environments.