Kafka connectors

Kafka Connectors

Lenses integrates with Kafka Connect to create, manage and monitor connectors via multiple connect clusters. In this guide you will teach you how to spin up and manage Kafka Connectors and monitor the generated flows.


Kafka Connect is one of the core APIs of Kafka ecosystem. Connectors are reusable components to bring data in and out of a Kafka cluster.

Required permission 

Kafka Connectors / ViewApplicationAllows viewing running Kafka Connectors based on your namespace
Kafka Connectors / ManageApplicationAllows add/update/delete/stop Kafka Connectors based on your namespace

When using Kafka Connect clusters you can authorise clusters per group. Select the approriate clusters per group to make it available to the users.

connect cluster permission

You can only use data within your data namespaces. If a topic is outside your namespace, you won’t be able to create a connector to either read or write to this topic.

How it works 

You can connect Lenses to one or multiple Kafka Connect clusters by configuring the cluster connection to Lenses config. For every cluster you will give an alias name, which will be used in Lenses, to help you navigate through the clusters where required.

To use Kafka Connectors, the cluster as well as the connector plugins need to be available. Connectors are made available via the cluster classpath and you can optionally add metadata to Lenses for better management.

When multiple clusters are available, you can use group management to authorize users for each cluster and control visibility.

View Connectors 

Once a cluster is made available in Lenses, all running connectors will be listed under the connectors list. You can have a quick overview per cluster and filter connector accordingly.

connectors list

Error management 

When a connector has a failure or is not running you will be able to preview it from the listing and drill down to connector details to view more details about the error as reported by the connector:

Connectors list task

The stacktrace exception can be returned by the connector:

connector detail task failing

Relevant metrics are also available for each task on your connector.

Manage connector 

You can start/stop the connector via the detailed view or edit the existing configuration. Lenses uses the native connect interfaces to apply the changes.

connector manage

Create Connector 

To create a new connector navigate to the Connectors list from the sidebar and select New Connector. You will be prompt with a list of connector plugins available in each cluster. Switch the cluster to have a look at the different connectors available:

new connector
new connector

The connector plugins are listed based on sources (bring data in Kafka) and sinks (bring data out of Kafka). Select the desired plugin and click to fill in the configuration details for the connector.

The mandatory fields will be prompted, you can select the optional fields to appear. The validation is running so that add all the required configurations:

new connector configuration

Sink connectors manage offsets 

For every sink connector the underlying consumer can be tracked. From there, you can manage the offsets ( ie. reset offsets or skip ) and also from the alert rules you can add consumer lag alerts for the relevant consumer. To edit the offsets your connector, therefore your consumer, should not be running ( aka committing offsets )

Connector sink consumer
Connector sink consumer detail

Custom connectors 

Any Kafka Connector that is following the Connect API standards, even if it is your custom connector, is compatible with Lenses, as long as it’s available in the cluster’s classpath and exposed via connect’s apis.

For better visibility, we advice to configure your connector’s details in Lenses so you can view its metadata.

View Connectors in Topology 

Your connectors instances are automatically plotted to the topology view of Lenses. From there you can see connectors and how they are linked with the rest of your streaming pipeline. Your topological view is also determined by your permissions.

Connectors topology

Supported connectors 

If you want to have better visibility of a 3rd party connector you can optionally enhance the information about it:

lenses {


  connectors.info = [
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"

View supported connectors:

lenses-cli connectors --supported

See also source connectors configuration.

Secret management 

External secret providers allow for indirect references to be placed in an application’s configuration so that secrets are not exposed in the Worker API endpoints of Kafka Connect.

API clients 

Connectors are also supported by the CLI to enable automation scenarios.



Does any Kafka Connector work with Lenses?

Yes! Lenses connects to the cluster and manages any available plugins.

See how to configure supported connectors metadata

Does Lenses.io offers Connectors for Kafka Connect?

Yes, Lenses.io is contributing to a number of Kafka Connectors open-sourced in the Kafka community. see the full list of supported connectors

Can we spin up new connect clusters?

You can connect all your connect clusters to Lenses, via Connections in the UI or automate via CLI/API, at any time through Lenses UI or Lenses CLI.

Can we have dynamically configuration of new connect clusters?

Yes. You can dynamically configure Lenses to connect to your Kafka Connect clusters at any time, through Lenses UI, Lenses CLI, or by using yaml files.

Can we have historic metrics of the connectors?

Not in Lenses per se. You can on the other hand use the Grafana templates that are provided by Lenses.io for historic metrics on your Kafka Connect

Can we get specific resource metrcis for each connector? ie. CPU, Memory etc

Not in Lenses per se. See our grafana dashboards

More resources