Kafka Connect
This page describes adding a Kafka Connect Cluster to the Lenses Agent.
Lenses integrates with Kafka Connect Clusters to manage connectors.
The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.
Multiple Kafka Connect clusters are supported.
If you are using Kafka Connect < 2.6 set the following to ensure you can see Connectors
lenses.features.connectors.topics.via.api.enabled=false
Consider Rate Limiting if you have a high number of connectors.
See JSON schema for support.
Environment variables are supported; escape the dollar sign
sslKeystorePassword:
value: "\${ENV_VAR_NAME}"
Names must match be alphanumeric or dash non-empty string.
Simple configuration, with JMX metrics
The URLs (workers) should always have a scheme defined (http:// or https://).
connect:
- name: my-connect-cluster-name
version: 1
tags: ["tag1"]
configuration:
workers:
value:
- http://my-kc.worker1:8083
- http://my-kc.worker2:8083
metricsPort:
value: 9585
metricsType:
value: JMX
Basic authentication
For Basic Authentication, define username
and password
properties.
connect:
- name: my-connect-cluster-name
tags: ["tag1"]
version: 1
configuration:
workers:
value:
- http://my-kc.worker1:8083
- http://my-kc.worker2:8083
username:
value: my-username
password:
value: my-password
TLS with custom truststore
A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.
connect:
- name: my-connect-cluster-name
tags: ["tag1"]
version: 1
configuration:
workers:
value:
- http://my-kc.worker1:8083
- http://my-kc.worker2:8083
sslTruststore:
file: /connect-truststore.jks
sslTruststorePassword:
value: myPassword
TLS with client authentication
A custom truststore might be necessary too (see above).
connect:
name: my-connect-cluster-name
tags: ["tag1"]
version: 1
configuration:
workers:
value:
- http://my-kc.worker1:8083
- http://my-kc.worker2:8083
sslKeystore:
file: connect-keystore.jks
sslKeystorePassword:
value: myPassword
Adding 3rd Party Connector to the Topology
If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info
parameter in the lenses.conf
file.
connectors.info = [
{
class.name = "The connector full classpath"
name = "The name which will be presented in the UI"
instance = "Details about the instance. Contains the connector configuration field which holds the information. If a database is involved it would be the DB connection details, if it is a file it would be the file path, etc"
sink = true
extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
icon = "file.png"
description = "A description for the connector"
author = "The connector author"
}
]
Last updated
Was this helpful?