Kafka Connect

Basic information 

Lenses uses Kafka Connect to manage connectors and SQL Processors (if using CONNECT mode).

Multiple Kafka Connect clusters might be used - a separate Connection should be created for each. Once a Kafka Connect connection is defined, Lenses displays all its workers and their statuses (with metrics, if configured so) in Workspace->Services screen.

Users need explicit permission via their groups to see a Connect cluster. In order to view, create, or manage a connector, they further need permissions to the topics it accesses.

The name of a Kafka Connect Connections may only contain alphanumeric characters ([A-Za-z0-9]) and dashes (-). Valid examples would be dev, Prod1, SQLCluster,Prod-1, SQL-Team-Awesome.

Metrics 

Metrics, if configured, are shown in each connector’s page and in the topology view. If some workers are ommited, the metrics will be incomplete, each worker only exports its own metrics.

SQL in Connect mode 

For running SQL Processors on a Connect cluster, provide metrics settings and an encryption key.

The AES-256 key along with the SQL connector installed in the Connect cluster are required to run SQL Processors in the cluster. The key’s length should be exactly 32 characters and it must match the key set in the Connect cluster.

Learn more about SQL in Connect .

Connection details 

Like other core services, Kafka Connect is managed via Connections. See more about managing connections here .

Multiple connections allowed: YES

Deletion allowed: YES

When managing connections other than with GUI, such information are relevant for Kafka Connect connection:

Template name: KafkaConnect

Examples 

The easiest way to manage Connections is via Lenses GUI under their respective pages, however it is also possible to do it directly via API, Helm or Lenses CLI. In such case, some connection type-specific values have to be used. Here are few examples of such configuration in YAML format.

• Find out more about managing Kafka Connect Connections via API
• Find out more about managing Connections via Lenses CLI provision
• Find out more about installing Lenses via Helm

Simple configuration, with JMX metrics 

The URLs (workers) should always have a scheme defined (http:// or https://).

AES-256 key is being used too, to make using SQL Processors in CONNECT mode possible.

  your-cluster-name:
    tags: [ "tag1" ]
    templateName: KafkaConnect
    configurationObject:
      workers:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083
      aes256Key: PasswordPasswordPasswordPassword
      # all metrics properties are optional
      metricsPort: 9581
      metricsType: JMX
      metricsSsl: false

Misc metrics configurations 

Find more about multiple options of configuring services’ metrics (like secured JMX, Jolokia, etc) under Services Metrics

Basic authentication 

For Basic Authentication, define username and password properties.

  your-cluster-name:
    tags: [ "tag1" ]
    templateName: KafkaConnect
    configurationObject:
      workers:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083
      username: my-username
      password: my-password

TLS with custom truststore 

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

  your-cluster-name:
    tags: [ "tag1" ]
    templateName: KafkaConnect
    configurationObject:
      workers:
        - https://my-kc.worker1:8083
        - https://my-kc.worker2:8083
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: myPassword

TLS with client authentication 

A custom truststore might be necessary too (see above).

  your-cluster-name:
    tags: [ "tag1" ]
    templateName: KafkaConnect
    configurationObject:
      workers:
        - https://my-kc.worker1:8083
        - https://my-kc.worker2:8083
      sslKeystore:
        fileRef:
          filePath: /path/to/my/keystore.jks
      sslKeystorePassword: keystorePassword
      sslKeyPassword: keyPassword
      

TLS with Basic Authentication 

As above - a custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

  your-cluster-name:
    tags: [ "tag1" ]
    templateName: KafkaConnect
    configurationObject:
      workers:
        - https://my-kc.worker1:8083
        - https://my-kc.worker2:8083
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: myPassword
      username: my-username
      password: my-password