Kafka Connections

Understand Kafka Connections and their usage in Lenses

Currently only used by Kafka to Kafka replicators

Kafka connections represent a set of credentials for a Kafka cluster in a Lenses environment. The credentials are never stored in Lenses; the connection holds a reference to Kubernetes Service Accounts or Secrets.

Applications consume the credentials from the Kubernetes secret or service account to establish a connection to the Kafka cluster. Lenses does not store secrets for the application connection.

A connection is attached to a Lenses Environment, but we do not create the credentials or Kubernetes resources. Each connection has:

1. Name

2. The name of the environment the credentials connect to (Kafka cluster)

3. The Kubernetes namespace the service accounts or secrets are in.

You are responsible for creating the Kubernetes Secrets, Service Accounts and the Kafka Users they hold the credentials for.

Prerequisites

1. The Source Kafka cluster (environment) where the application will connect to, must have network access from the Kubernetes cluster. You can override the brokers configuration for the selected environment, if required.

2. You are responsible for creating the Kafka user (credentials) for the source Kafka cluster.

3. You are responsible for creating the Kubernetes secret or service account that contains the credentials for the application to connect to the source Kafka cluster.

4. AWS MSK clusters, using AWS IAM authentication, require the creation of an Kubernetes service account, with the necessary AWS IAM permissions to access the Kafka cluster.

5. Applications using this connection must be deployed in the same Kubernetes namespace as the secrets or service accounts to access them.

AWS MSK IAM

6.1 currently supports service accounts. Later releases will support other authentication methods.

Kubernetes service accounts are used to support connections to AWS MSK IAM. A connection must be created with the name of a service account that has the required AWS IAM policies to connect to your AWS MSK cluster.

Creating a Kafka Connection

Select the Apps option from the left sidebar, either open the listing for Kafka Connectors and click Create Kafka Connection, or click create from the node in the tree panel.

1

Provide a name

Give the connection a name and description

2

Select the Kafka Cluster to connect to

Select the environment to which the credentials (in the Kubernetes Secret or Service Account) connects to.

3

Select the deployment environment

This is the environment the application consuming the secrets or service accounts are deployed to. Its also the environment Kafka connection is attached to.Give the connection a name and description

Connections for Kafka to Kafka Replication

To enable a route for Kafka to Kafka, you must create two Kafka connections, one for each Kafka environment. Both connections must have the same deployment environment (including namespace).

For AWS MSK IAM, the deployment environment service account must have policies attached for both the environments the connections connect to.