Kafka Connect

This page describes adding a Kafka Connect Cluster to the Lenses Agent.

Lenses integrates with Kafka Connect Clusters to manage connectors.

The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Multiple Kafka Connect clusters are supported.

If you are using Kafka Connect < 2.6 set the following to ensure you can see Connectors

lenses.features.connectors.topics.via.api.enabled=false

Consider Rate Limiting if you have a high number of connectors.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  value: "\${ENV_VAR_NAME}"

Simple configuration, with JMX metrics

The URLs (workers) should always have a scheme defined (http:// or https://).

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    version: 1    
    tags: ["tag1"]
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083
      metricsPort: 
        value: 9585
      metricsType: 
        value: JMX              

Basic authentication

For Basic Authentication, define username and password properties.

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      username: 
        value: my-username
      password: 
        value: my-password

TLS with custom truststore

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

provisioning.yaml

connect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      sslTruststore:
        file: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

provisioning.yaml

connect:
  name: my-connect-cluster-name
  tags: ["tag1"]
  version: 1    
  configuration:
    workers:
      value:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083    
    sslKeystore:
      file: /path/to/my/keystore.jks
    sslKeystorePassword: 
      value: myPassword

Adding 3rd Party Connector to the Topology

If you have developed your own Connector or are using not using a Lenses connector you can still display the connector instances in the topology. To do this Lenses needs to know the configuration option of the Connector that defines which topic the Connector reads from or writes to. This is set in the connectors.info parameter in the lenses.conf file.

lenses.conf

connectors.info = [
      {
           class.name = "The connector full classpath"
           name = "The name which will be presented in the UI"
           instance = "Details about the instance. Contains the connector configuration field which holds the information. If  a database is involved it would be  the DB connection details, if it is a file it would be the file path, etc"
           sink = true
           extractor.class = "The full classpath for the implementation knowing how to extract the Kafka topics involved. This is only required for a Source"
           icon = "file.png"
           description = "A description for the connector"
           author = "The connector author"
      }

  ]