Integrating data with Kafka Connect

Understand how to create and manage Kafka Connectors in Lenses

For the available Lenses Apache 2.0 connectors, see the Stream Reactor documentation.

Lenses connects to Kafka Connect clusters through the Kafka Connect REST API. You can deploy connectors outside Lenses and still use Lenses to inspect and manage them.

Add one or more Kafka Connect clusters to the agent provisioning file. Once connected, Lenses discovers the installed connector plugins on each cluster.

To update the provisioning file, open Environments from the left navigation, expand the target environment, then open Configuration. From there you can edit, test, and apply the configuration through the connected agent.

Create a Kafka Connector

Open Apps from the left navigation. Then open Kafka Connectors and select Create Kafka Connector. You can also create one from the Kafka Connectors node in the navigation tree.

Add metadata and select the cluster

Enter the connector name, select the environment, then select the Kafka Connect cluster. The editor then becomes active and starts offering auto-complete for the available connector plugins.

Select the Connector

Choose the connector plugin. For sink connectors, the editor then moves to the topics field and offers auto-complete for available topics in the cluster.

Use Ctrl + Space or Cmd + Space to open more detail in the editor auto-complete menu.

In the editors, use Ctrl + Space or Cmd + Space for auto-complete and default snippets.

Task Max

Enable task max to let the editor calculate tasks.max from the selected topic partition counts.

Initial State at Creation

Set the connector's initial state with the stop, pause, and play icons.

Validation & Deployment

Use Validate to test the configuration against the Kafka Connect cluster. Validation errors appear in the editor and problem panel.

Use Deploy to create the connector on the Kafka Connect cluster.

The connector is deployed with the selected initial state. By default, that state is stopped.

View and manage connectors

Open Apps from the left navigation, then open Kafka Connectors. You can also expand a target environment and open its connector nodes there.

Select a connector from the list or from the navigation tree to open its details.

Each connector has these sections. You can open them from breadcrumbs, from the navigation tree, or from context menus.

Profile — view metadata, summary metrics, and the topics involved. Topic entries include context actions for deeper inspection.
Configuration — update the connector configuration. You cannot change the connector plugin type after creation.
Health — inspect task health, metrics, and stack traces.
Alerts — view alert events and manage alert rules. See Alerting & Monitoring.
Activity — inspect the audit trail for the connector. See Governance.