Install

Configure HQ

HQ is configured via by one file, config.yaml. The docker compose files loads the content of hq.config.yaml and mounts it as the HQ config.yaml file.

Adding a Database Connection

Edit the docker-compose.yaml and add the set the credentials for your database in the hq.config.yaml section.

 hq.config.yaml:
    content: |
      # ACCEPT THE LENSES EULA
      license:
        acceptEULA: true
      database:
        host: postgres:5432
        username: [YOUR_POSTGRES_YOUR_NAME]
        password: lenses
        database: hq

Authentication

Currently HQ supports:

1. Basic Authentication (default)

2. SAML

For this example we will use basic authentication, for information on configuring other methods, see Authentication and configure the hq.config.yaml key accordingly for SAML.

Start HQ

To start HQ, run the following docker command:

docker-compose up hq

You can now log in your browser with admin/admin.

Create an Environment for your Kafka Cluster

To create an environment in HQ:

1. Login into HQ and create an environment, Environments->New Environment.

2. At the end of the process, you will be shown an Agent Key. Copy that, keep it safe!

The environment will be disconnected until the Agent is up and configured with the key.

You can also manage environments using the CLI.

➜  out ./hq-linux-amd64 environments
Manage Environments.

Usage:
  hq environments [command]

Aliases:
  environments, e, env, envs

Available Commands:
  create      Creates a new environment.
  delete      Deletes an environment.
  get         Retrieves a single environment by name.
  list        Lists all environments
  metadata    Manages environment metadata.
  update      Updates an environment.
  watch       Watch live environment updates.

Configure the Agent

The Agent is configured via two files:

• lenses.conf - holds low-level configuration options for the agent and the database connection. You can set this via the agent.lenses.conf in the docker-compose file

• provisioning.yaml - holds the connection details to your Kafka cluster and supporting systems. can set this via the agent.provisioning.yaml key in the docker-compose file.

Adding an Agent Database Connection

Update the docker-compose file agent.lenses.conf key for your Postgres instance.

 agent.lenses.conf:
    content: |
      lenses.storage.postgres.host=[YOUR_POSTGRES_INSTANCE]
      lenses.storage.postgres.port=[YOUR_POSTGRES_PORT]
      lenses.storage.postgres.database=agent
      lenses.storage.postgres.username=lenses
      lenses.storage.postgres.password=lenses

Adding a HQ Connection

The Agent Key for an environment needs to be added to the agent.provisioning.yaml key in the docker compose file.

agent.provisioning.yaml:
    content: |
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace ${{LENSESHQ_AGENT_KEY}} with the Agent Key for the environment that you want to link to.

For more information on the configuration of the connection to HQ see here.

Adding a Kafka Connection

By default, the agent is configured to connect to Kafka on localhost. To change this update the agent.provisioning.yaml key. The information required here depends on how you want the Agent to authenticate against Kafka.

1. Add the following for a basic plaintext connection to a Kafka broker, if you are using a different authentication mechanism adjust accordingly.

2. Remove, or adjust the Kafka (kafka-demo), Schema Registry and Connect services in the default docker-compose file.

agent.provisioning.yaml:
    content: |
      kafka:
        - name: kafka
          version: 1
          tags: [ 'kafka', 'dev' ]
          configuration:
            metricsType:
              value: JMX
            metricsPort:
              value: 9581
            kafkaBootstrapServers:
              value: PLAINTEXT://[YOUR_BOOTSTRAP_BROKER:PORT] 
            protocol:
              value: PLAINTEXT
      lensesHq:
        - configuration:
            agentKey:
              value: $${LENSESHQ_AGENT_KEY}
            port:
              value: 10000
            server:
              value: lenses-hq
          name: lenses-hq
          tags: ['hq']
          version: 1

Replace [YOUR_BOOTSTRAP_BROKER:PORT] with the bootstrap brokers and ports for the Kafka cluster you want the Agent to connect to.

Start the Agent

To start Agent, run the following docker command:

docker-compose up agent

Once the agent fully starts, it will report as connected in HQ, allowing you to explore your Kafka environments.