4.0

You are viewing documentation for an older version of Lenses.io View latest documentation here

Kafka Connect

Add one or more Kafka Connect cluster by providing a name to distinguish it and all the worker urls (including protocol) and the three backing kafka-connect cluster topics (connect-status, connect-configs and connect-offsets).

Authentication and metrics can be configured as well (for details - check proper tab).

One Kafka Connect cluster with two workers:

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     { url:"http://CONNECT_HOST_1:8083" },
     { url:"http://CONNECT_HOST_2:8083" }
   ],
   statuses: "connect-status",
   configs : "connect-configs",
   offsets : "connect-offsets"
 }
]
One Kafka Connect cluster with two workers using basic authentication.

For Aiven you must use the secure https protocol.

lenses.kafka.connect.clusters = [
  {
    name: "change_data_capture",
    username: "USERNAME",
    password: "PASSWORD",
    auth: "USER_INFO",
    urls: [
      // non-empty list of workers urls
    ],
    statuses: "connect-status",
    configs: "connect-configs",
    offsets: "connect-offsets"
  }
]

One Kafka Connect cluster with two workers and JMX metrics.

When defining metrics objects, particular entries should have the same values of type field. Otherwise, the type of first metrics object is treatened as the valid one for all metrics.

In principle - the host part of metrics.url key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below) should be the same as the worker url host itself.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     { url:"http://CONNECT_HOST_1:8083",
       metrics: {
          type: "JMX",               
          url: "CONNECT_HOST_1:9584" # no scheme should be included, just host and port
       }
     { url:"http://CONNECT_HOST_2:8083",
       metrics: {
          type: "JMX",               
          url: "CONNECT_HOST_2:9584" # no scheme should be included, just host and port
       }
   ],
   statuses: "connect-status",
   configs : "connect-configs",
   offsets : "connect-offsets"
 }
]

One Kafka Connect cluster with two workers and authenticated JMX metrics over SSL.

Particular metrics entries should have the same values of following fields: type, ssl, user, password.
Otherwise, the first metrics object is treatened as the valid one for those keys.

In principle - the host part of metrics.url key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below) should be the same as the worker url host itself.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     {
       url:"http://CONNECT_HOST_1:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # Optional (default: false); if used - please make sure the remote 
                            # certificate is accepted by Lenses truststore
         user: "admin",     # JMX protected by user/pass
         password: "admin",
         type: "JMX",       
         url: "CONNECT_HOST_1:9584" # no scheme should be included, just host and port
       }
     },
     {
       url:"http://CONNECT_HOST_2:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # Optional (default: false); if used - please make sure the remote 
                            # certificate is accepted by Lenses truststore
         user: "admin",     # JMX protected by user/pass
         password: "admin",
         type: "JMX",       
         url: "CONNECT_HOST_2:9584" # no scheme should be included, just host and port
       }
     }
   ],
   statuses: "connect-status",
   configs: "connect-configs",
   offsets: "connect-offsets"
 }
]

One Kafka Connect cluster with two workers and authenticated Jolokia metrics over SSL.

Particular metrics entries should have the same values of following fields: type, ssl, user, password.
Otherwise, the first metrics object is treatened as the valid one for those keys.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     {
       url:"http://CONNECT_HOST_1:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # Optional (default: false); if used - please make sure the remote 
                            # certificate is accepted by Lenses truststore
         user: "",          # Optional, the remote HTTP user
         password: "",      # Optional, the remote HTTP password
         type: "JOLOKIAP",     # One of 'JOLOKIAG' (for GET requests), 'JOLOKIAP' (for POST requests)
         url: "https://host1:9585/jolokia/" # notice the scheme and trailing slash - they are both required for standard Jolokia configuration
       }
     },
     {
       url:"http://CONNECT_HOST_2:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # Optional (default: false); if used - please make sure the remote 
                            # certificate is accepted by Lenses truststore
         user: "",          # Optional, the remote HTTP user
         password: "",      # Optional, the remote HTTP password
         type: "JOLOKIAP",     # One of 'JOLOKIAG' (for GET requests), 'JOLOKIAP' (for POST requests)
         url: "https://host2:9585/jolokia/" # notice the scheme and trailing slash - they are both required for standard Jolokia configuration
       }
     }
   ],
   statuses: "connect-status",
   configs: "connect-configs",
   offsets: "connect-offsets"
 }
]

Kafka Connect SSL/TLS Http configuration.

Particular metrics entries should have the same values of following fields: type, ssl, user, password.
Otherwise, the first metrics object is treatened as the valid one for those keys.

In principle - the host part of metrics.url key (“CONNECT_HOST_1” & “CONNECT_HOST_2” in examples below) should be the same as the worker url host itself.

lenses.kafka.connect.clusters = [
 {
   name: "change_data_capture",
   urls: [
     {
       url:"http://CONNECT_HOST_1:8083",
       metrics: {           # Metrics section is (optional)
         ssl: true,         # SSL - ensure JMX/HTTP certificate is accepted by Lenses truststore
         user: "admin",     # JMX protected by user/pass
         password: "admin",
         type: "JMX",       # One of 'JMX', 'JOLOKIAP' (POST), 'JOLOKIAG' (GET)
         url: "CONNECT_HOST_1:9584" # JMX port is 9584; Scheme (http or https) is required when using JOLOKIAP and JOLOKIAG !
       }
     }
   ],
   statuses: "connect-status",
   configs: "connect-configs",
   offsets: "connect-offsets"
 }
]
lenses.kafka.connect.ssl.keystore.location   = "/path/to/keystore.jks"
lenses.kafka.connect.ssl.keystore.password   = "changeit"
lenses.kafka.connect.ssl.key.password        = "changeit"
lenses.kafka.connect.ssl.truststore.location = "/path/to/truststore.jks"
lenses.kafka.connect.ssl.truststore.password = "changeit"

See configuration settings.