Kafka Connections

This page describes how to manage Kafka Connections in Lenses.

Currently only used by Kafka to Kafka replicators

Kafka connections represent a set a of credentials for a Kafka cluster in a Lenses environment. The credentials are never stored in Lenses, the connection holds a reference to Kubernetes Service Accounts or Secrets.

Applications consume the credentials from the Kubernetes secret or service account to establish a connection to the Kafka cluster. Lenses does not store secrets for the application connection.

A connection is attached a Lenses Environment, but we do not create the the credentials or Kubernetes resources. Each connection has

1. Name

2. The name of the environment the credentials connect to (Kafka cluster)

3. The Kubernetes namespace the service accounts or secrets are in.

You are responsible for creating the Kubernetes Secrets, Service Accounts and the Kafka Users they hold the credentials for.

Prerequisites

1. The Source Kafka cluster (environment) where the application will connect to, must have network access from the Kubernetes cluster. You can override the brokers configuration for the selected environment, if required.

2. You are responsible for creating the Kafka user (credentials) for the source Kafka cluster.

3. You are responsible for creating the Kubernetes secret or service account that contains the credentials for the application to connect to the source Kafka cluster.

4. AWS MSK clusters, using AWS IAM authentication, require the creation of an Kubernetes service account, with the necessary AWS IAM permissions to access the Kafka cluster.

5. Applications using this connection must be deployed in the same Kubernetes namespace as the secrets or service accounts to access them.

AWS MSK IAM

6.1 currently supports service accounts. Later releases will support other authentication methods.

Kubernetes service accounts are used to support connections to AWS MSK IAM. A connection must be created with the name of a service account that has the required AWS IAM policies to connect to your AWS MSK cluster. To set up an IAM policies see here.

Creating a Kafka Connection

Go to Apps, New, Kafka Connection.

1

Select the Kafka Cluster to connect to

Select the environment to which the credentials (in the Kubernetes Secret or Service Account) connects to.

2

Select the deployment environment

This is the environment the application consuming the secrets or service accounts are deployed to. Its also the environment Kafka connection is attached to.

Connections for Kafka to Kafka Replication

To enable a route for Kafka to Kafka, you must create two, Kafka connections, one for each Kafka environment. Both connections must have the same deployment environment (including namespace).

For AWS MSK IAM, the deployment environment service account, must have policies attached for both the environments the connections connect to.