Kafka Connect

This page describes adding a Kafka Connect Cluster to Lenses.

Lenses integrates with Kafka Connect Clusters to manage connectors.

For documentation about the available Lenses Apache 2.0 Connectors, see the Stream Reactor documentation.

The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Multiple Kafka Connect clusters are supported.

If you are using Kafka Connect < 2.6 set the following to ensure you can see Connectors

lenses.features.connectors.topics.via.api.enabled=false

See provisioning for automating connections.

Consider Rate Limiting if you have a high number of connectors.

Adding a connection

To add a connection, go to Admin->Connections->New Connection->Kafka Connect.

1. Provide a name for the Connect cluster

2. Add a comma-separated list of the workers in the Connector cluster, including ports

3. Optionally enable Basic Auth and set the username and password

4. Optionally enable SSL and upload the key-store file

5. Optionally upload a trust store

6. Optionally enable the collection of JMX metrics (Simple or Jolokia with SSL and Basic auth support)

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]