4.0

You are viewing documentation for an older version of Lenses.io View latest documentation here

Kafka Connect

Add one or more Kafka Connect cluster by providing a name to distinguish it and all the worker urls (including protocol) and the three backing kafka-connect cluster topics (connect-status, connect-configs and connect-offsets).

Authentication and metrics can be configured as well (for details - check proper tab).

One Kafka Connect cluster with two workers:

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     { url:"http://CONNECT_HOST_1:8083" },
     { url:"http://CONNECT_HOST_2:8083" }
   ],
   statuses: "connect-status",
   configs : "connect-configs",
   offsets : "connect-offsets"
 }
]
One Kafka Connect cluster with two workers using basic authentication.

For Aiven you must use the secure https protocol.

lenses.kafka.connect.clusters = [
  {
    name: "change_data_capture",
    username: "USERNAME",
    password: "PASSWORD",
    auth: "USER_INFO",
    urls: [
      // non-empty list of workers urls
    ],
    statuses: "connect-status",
    configs: "connect-configs",
    offsets: "connect-offsets"
  }
]

One Kafka Connect cluster with two workers and JMX metrics.

When defining metrics objects, particular entries should have the same values of type field. Otherwise, the type of first metrics object is treatened as the valid one for all metrics.

In principle - the host part of metrics.url key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below) should be the same as the worker url host itself.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     { url:"http://CONNECT_HOST_1:8083",
       metrics: {
          type: "JMX",               # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
          url: "CONNECT_HOST_1:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
       }
     { url:"http://CONNECT_HOST_2:8083",
       metrics: {
          type: "JMX",               # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
          url: "CONNECT_HOST_2:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
       }
   ],
   statuses: "connect-status",
   configs : "connect-configs",
   offsets : "connect-offsets"
 }
]

One Kafka Connect cluster with two workers and authenticated JMX metrics over SSL.

Particular metrics entries should have the same values of following fields: type, ssl, user, password.
Otherwise, the first metrics object is treatened as the valid one for those keys.

In principle - the host part of metrics.url key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below) should be the same as the worker url host itself.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     {
       url:"http://CONNECT_HOST_1:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # SSL - ensure JMX/HTTP certificate is accepted by Lenses truststore
         user: "admin",     # JMX protected by user/pass
         password: "admin",
         type: "JMX",       # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
         url: "CONNECT_HOST_1:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
       }
     },
     {
       url:"http://CONNECT_HOST_2:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # SSL - ensure JMX/HTTP certificate is accepted by Lenses truststore
         user: "admin",     # JMX protected by user/pass
         password: "admin",
         type: "JMX",       # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
         url: "CONNECT_HOST_2:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
       }
     }
   ],
   statuses: "connect-status",
   configs: "connect-configs",
   offsets: "connect-offsets"
 }
]

Kafka Connect SSL/TLS Http configuration.

Particular metrics entries should have the same values of following fields: type, ssl, user, password.
Otherwise, the first metrics object is treatened as the valid one for those keys.

In principle - the host part of metrics.url key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below) should be the same as the worker url host itself.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     {
       url:"http://CONNECT_HOST_1:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # SSL - ensure JMX/HTTP certificate is accepted by Lenses truststore
         user: "admin",     # JMX protected by user/pass
         password: "admin",
         type: "JMX",       # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
         url: "CONNECT_HOST_1:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
       }
     }
   ],
   statuses: "connect-status",
   configs: "connect-configs",
   offsets: "connect-offsets"
 }
]
lenses.kafka.connect.ssl.keystore.location   = "/path/to/keystore.jks"
lenses.kafka.connect.ssl.keystore.password   = "changeit"
lenses.kafka.connect.ssl.key.password        = "changeit"
lenses.kafka.connect.ssl.truststore.location = "/path/to/truststore.jks"
lenses.kafka.connect.ssl.truststore.password = "changeit"

See configuration settings.