Kafka Connect

Kafka Connect

Lenses integrates with multiple connect clusters. You can add them to specific groups and allow users to create, manage or monitor connectors.

Create Kafka Connect Connection 

You can create a Kafka Connect Connection with Lenses by navigating to Admin and then from the left hand menu select Connections.

Kafka connect create Connection

Then from the connection options select Kafka Connect:

Kafka Connect selection

Then fill in your Kafka Connect connection details and click on Test Kafka Connect Connection to verify:

Kafka connect create Connection details

After the connection is made succesfully, see the user’s guide on how to manage connectors.

Here is an example:

lenses.kafka.connect.clusters = [
   name: "clusterA",
   urls: [
     { url:"http://CONNECT_HOST_A_1:8083" },
     { url:"http://CONNECT_HOST_A_2:8083" }
   statuses: "connect-status-a",
   configs : "connect-configs-a",
   offsets : "connect-offsets-a"
   name: "clusterB",
   urls: [
     { url:"http://CONNECT_HOST_B_1:8083" },
     { url:"http://CONNECT_HOST_B_2:8083" }
   statuses: "connect-status-b",
   configs : "connect-configs-b",
   offsets : "connect-offsets-b"

Review Kafka Connect configuration options

Cluster Names

Name is a short alias for the cluster to use with Lenses. The name of the Kafka Connect cluster (lenses.kafka.connect.clusters[].name) may only contain alphanumeric characters ([A-Za-z0-9]) and dash (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Kafka connect services

Authorize groups 

Lenses integrates with the native connect APIs and adds group governance on top.

Kafka connect groups

Restarting the Lenses application with different lenses.kafka.connect.clusters mean the following:

  1. If any cluster was removed - all the groups will lose permission to removed cluster.
  2. If any cluster was added - the cluster won’t be visible to any groups, permissions have to be granted.

Multiple clusters 

Multiple Kafka Connect clusters might be used - a separate Connection should be created for each. Once a Kafka Connect connection is defined, Lenses displays all its workers and their statuses (with metrics, if configured so) in Workspace->Services screen.

Users need explicit permission via their groups to see a Connect cluster. In order to view, create, or manage a connector, they further need permissions to the topics it accesses.

Supported connectors 

You can select a supported connector from LensesUI:

Kafka connect select

See the full list of Lenses supported connectors.