Integrating data with Kafka Connect

Understand how to create and manage Kafka Connectors in Lenses

For the available Lenses Apache 2.0 connectors, see the Stream Reactor documentation.

Lenses connects to Kafka Connect clusters through the Kafka Connect REST API. You can deploy connectors outside Lenses and still use Lenses to inspect and manage them.

Add one or more Kafka Connect clusters to the agent provisioning file. Once connected, Lenses discovers the installed connector plugins on each cluster.

To update the provisioning file, open Environments from the left navigation, expand the target environment, then open Configuration. From there you can edit, test, and apply the configuration through the connected agent.

Create a Kafka Connector

Open Apps from the left navigation. Then open Kafka Connectors and select Create Kafka Connector. You can also create one from the Kafka Connectors node in the navigation tree.

1

Add metadata and select the cluster

Enter the connector name, select the environment, then select the Kafka Connect cluster. The editor then becomes active and starts offering auto-complete for the available connector plugins.

2

Select the Connector

Choose the connector plugin. For sink connectors, the editor then moves to the topics field and offers auto-complete for available topics in the cluster.

Use Ctrl + Space or Cmd + Space to open more detail in the editor auto-complete menu.

The create connector flow includes these tabs:

1. Config — the main connector configuration. Lenses builds a schema from the connector plugin definition, and this is where most settings live.

2. Converters — converter settings inferred from the selected topics and formats. For example, Avro topics can trigger Avro converter defaults.

3. SMTs — single message transforms available on the Kafka Connect cluster, including templates and schema-backed editing.

4. Consumer & Producer Overrides — per-connector overrides with templates and schema-backed editing.

5. Error Handling — adds Kafka Connect error handling defaults. If the selected Lenses connector supports extra connector-level error handling, those settings are added too.

6. Manifest — the final read-only configuration that Lenses sends to Kafka Connect.

In the editors, use Ctrl + Space or Cmd + Space for auto-complete and default snippets.

Task Max

Enable task max to let the editor calculate tasks.max from the selected topic partition counts.

Initial State at Creation

Set the connector's initial state with the stop, pause, and play icons.

Validation & Deployment

Use Validate to test the configuration against the Kafka Connect cluster. Validation errors appear in the editor and problem panel.

Use Deploy to create the connector on the Kafka Connect cluster.

The connector is deployed with the selected initial state. By default, that state is stopped.

View and manage connectors

Open Apps from the left navigation, then open Kafka Connectors. You can also expand a target environment and open its connector nodes there.

Select a connector from the list or from the navigation tree to open its details.

Each connector has these sections. You can open them from breadcrumbs, from the navigation tree, or from context menus.

1. Profile — view metadata, summary metrics, and the topics involved. Topic entries include context actions for deeper inspection.

2. Configuration — update the connector configuration. You cannot change the connector plugin type after creation.

3. Health — inspect task health, metrics, and stack traces.

4. Alerts — view alert events and manage alert rules. See Alerting & Monitoring.

5. Activity — inspect the audit trail for the connector. See Governance.

Start, stop, restart, or delete connectors and tasks

Use the connector tab context menu or the connector node in the navigation tree to manage the connector and its tasks.

Common actions include:

1. start a stopped connector

2. pause or stop a running connector

3. restart a connector or an individual task

4. delete a connector

Stopping, restarting, or deleting a connector interrupts active ingestion or delivery. Confirm the target connector and cluster before you proceed.

Related pages

Use these pages next:

1. Sources — available source connectors

2. Sinks — available sink connectors

3. Secret Providers — manage connector secrets securely