1 of 12

Automating Connections

This page describes automating (provisioning) connections and channels for Lenses at installation and how to apply updates.

On start-up, Lenses will be in bootstrap mode unless it has an existing Kafka Connection.

To fully start Lenses you need two key pieces of information to start and perform basic functions:

Kafka Connection
Valid License

Provisioning

If provisioning is enabled, any changes in the UI will be overriden.

A dedicated API, called provisioning, is available to handle bootstrapping key connections at installation time. This allows you to fully install and configure key connections such as Kafka, Schema Registry, Kafka Connect, and Zookeepers in one go. You can use either of the following approaches depending on your needs:

Both approaches use a YAML file to define connections.

Defining a Connection

Connections are defined in theprovisioning.yaml. This file is divided into components, each component representing a type of connection.

Each component must have:

Name - This is the free name of the connection
Version set to 1
Optional tags
Configuration - This is a list of keys/values and is dependent on the component type.

For a full list of configuration options for the connect see Provisioning API Spec.

File Watcher Provisioning

This page describes how to use the Lenses File Watcher to setup connections to Kafka and other services and have changes applied.

Connections are defined in the provisioning.yaml file. Lenses will then watch the file and resolve the desired state, applying connections defined in the file.

If a connection is not defined but exists in Lenses it will be removed. It is very important to keep your provision YAML updated to reflect the desired state.

Enabling File Watcher Provisioning

File watcher provisioning must be explicitly enabled. Set the following in the lenses.conf file:

lenses.conf

# Directory containing the provision.yaml files
lenses.provisioning.path=/my/dir
# The interval at which Lenses will check for updates 
# in the file seconds
lenses.provisioning.interval=10s

Updates to the file will be loaded and applied if valid without a restart of Lenses.

Directory layout

Lenses expects a set of files in the directory, defined by lenses.provisioning.path. The structure of the directory must follow:

files/ directory for storing any certificates, JKS files or other files needed by the connection
provisioning.yaml - This is the main file, holding the definition of the connections
license.json - Your lenses license file

➜  ~ tree provisioning-folder
provisioning-folder
├── files
│   └── tuststore.jks
├── license.json
└── provisioning.yaml

4 directories, 0 files
➜  ~

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm, the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime.

values.yaml

sslKeystorePassword:
  value: ${ENV_VAR_NAME}

Referencing files

Many connections need files, for example, to secure Kafka with SSL you will need a key store and optionally a trust store.

To reference a file in the provisioning.yaml, for example, given:

values.yaml

    configuration:
      protocol:
        value: SASL_SSL
      sslKeystore:
        file: "my-keystore.jks"

a file called my-keystore.jks is expected in the files directory. This file will be used for the key store location.

API Provisioning

This page describes how to use the Lenses provisioning API to setup connections to Kafka and other services and have changes applied.

Building on the provisioning.yaml, API provisiong allows for uploading the files directly Lenses from anywhere with network access and without access to the host where Lenses is installed.

Uploading supporting files

Many connections need files, for example, to secure Kafka with SSL you will need a keystore and optionally a trust store.

To reference a file in the, for the configuration option set the key to be "file" and the value to reference in the API request. For example, given:

    configuration:
      protocol:
        value: SASL_SSL
      sslKeystore:
        file: "my-keystore-file"

To upload the file to be used for the configuration option sslKeystore: add the following to the request:

--form "my-keystore-file=@${PATH_TO_KEYSTORE_FILE};type=application/octet-stream" \

API Call

Set the type to application/octet-stream.
The name of the part in the multipart request (supporting files) should match the value of the property pointing to the mounted file in the provisioning.yaml descriptor. This ensures accurate mapping and referencing of files.
Set LENSES_SESSION_TOKEN as the value of the Lenses Service Account token you want to use to automate provisioning.

curl --location --request PUT "${LENSES_ENV}/api/v1/state/connections" \
   --header "X-Kafka-Lenses-Token: ${LENSES_SESSION_TOKEN}" \
   --header 'Content-Type: multipart/form-data' \
   --header 'Content-Disposition: form-data;' \
   --form "my-keystore-file=@${PATH_TO_KEYSTORE_FILE};type=application/octet-stream" \
   --form 'provisioning=@"resources/provisioning.yaml";type=text/plain(utf-8)'

In this example, the provisioning.yaml is read from provisioning=@"resources/provisioning.yaml.

The provisioning.yaml contains a reference to "my-keystore-file" which is loaded from @${PATH_TO_KEYSTORE_FILE};type=application/octet-stream

Managing secrets

The provisioning.yaml contains secrets. If you are deploying via Helm the chart will use Kubernetes secrets.

Additionally, support is provided for referencing environment variables. This allows you to set secrets in your environment and have the value resolved at runtime. i.e. inject an environment variable from GitHub secrets for passwords.

sslKeystorePassword:
  value: ${ENV_VAR_NAME}

Provisioning examples

This page gives examples of the provisioning yaml for Lenses.

To use with Helm file place the examples under lenses.provisioning.connections in the values file.

Kafka

This page provides examples for defining a connection to Kafka.

If deploying with Helm put the connections YAML under provisioning in the values file.

PLAINTEXT

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.
protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword

SASL_PLAINTEXT vs SASL_SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is if:

The transport layer is encyrpted (SSL)
The SASL mechanisn for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

Apart from that, when encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

Following are a few examples of SASL_PLAINTEXT and SASL_SSL

SASL_SSL

PLAIN

Encrypted communication and basic username and password for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";

AWS_MSK_IAM

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

GSSAPI

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword  
    saslMechanism: 
      value: GSSAPI
    saslJaasConfig:
      value: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";      
     keytab:
       file: /path/to/keytab.jks

SASL_PLAINTEXT

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

SCRAM-SHA-256

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

SCRAM-SHA-512

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

Advanced Client Configuration

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configurationObject:
    kafkaBootstrapServers:
      value:
       - PLAINTEXT://your.kafka.broker.0:9092
    protocol: 
      value: PLAINTEXT
    additionalProperties:
      value:
        isolation.level: "read_committed"
        acks: "all"
        ssl.endpoint.identification.algorithm: "https"

Zookeeper

This page provides examples for defining a connection to Zookeeper.

Simple configuration, without metrics

Simple configuration, with JMX metrics

Simple configuration with Zookeeper metrics read via JMX.

With such a configuration, Lenses will use 3 Zookeeper nodes and will try to read their metrics from following URLs (notice the same port - 9581 - used for all of them, as defined by metricsPort property):

my-zookeeper-host-0:9581
my-zookeeper-host-1:9581
my-zookeeper-host-2:9581

Schema Registry

This page provides examples for defining a connection to Schema Registries.

Confluent

Simple configuration, with JMX metrics

The URLs (nodes) should always have a scheme defined (http:// or https://).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      ## all metrics properties are optional
      metricsPort: 
        value: 9581
      metricsType: 
        value: JMX
      metricsSsl: 
        value: false

Basic authentication

For Basic Authentication, define username and password properties.

confluentSchemaRegistry:
- name: schema-registry
  tags: ["tag1"]
  version: 1    
  configuration:
    schemaRegistryUrls:
      value:
        - http://my-sr.host1:8081
        - http://my-sr.host2:8081
    username: 
      value: my-username
    password: 
      value: my-password

TLS with custom truststore

A custom truststore is needed when the Schema Registry is served over TLS (encryption-in-transit) and the Registry’s certificate is not signed by a trusted CA.

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslTruststore:
        fileRef:
          filePath: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      sslKeystore:
        fileRef:
          filePath: /path/to/my/keystore.jks
      sslKeystorePassword: 
        value: myPassword

Hard or soft delete

By default, Lenses will use hard delete for Schema Registry. To use soft delete, add the following property:

confluentSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      schemaRegistryUrls:
        value:
          - http://my-sr.host1:8081
          - http://my-sr.host2:8081
      hardDelete:
        value: true

AWS Glue

Some connections depend on others. One example is the AWS Glue Schema Registry connection, which depends on an AWS connection. These are examples of provision Lenses with an AWS connection named my-aws-connection and an AWS Glue Schema Registry that references it.

Using AWS Access Key

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Access Key
      accessKeyId: 
        value: my-access-key-id
      secretAccessKey: 
        value: my-secret-access-key
      region: 
        value: eu-west-1
      
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: my-aws-connection
      accessKeyId:
        reference: my-aws-connection
      secretAccessKey:
        reference: my-aws-connection
      glueRegistryArn:
          value: arn:aws:glue:region:123123123:registry/my-registry

Using AWS Credentials Chain

aws:
  - name: my-aws-connection
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: Credentials Chain
      region: 
        value: eu-west-1
        
glueSchemaRegistry:
  - name: schema-registry
    tags: ["tag1"]
    version: 1      
    configuration:
      authMode: 
        value: my-aws-connection
      glueRegistryArn:
        value: arn:aws:glue:region:123123123:registry/my-registry

Kafka Connect

This page provides examples for defining a connection to Kafka Connect Clusters.

Simple configuration, with JMX metrics

The URLs (workers) should always have a scheme defined (http:// or https://).

This example uses an optional AES-256 key. The key decodes values encoded with AES-256 to enable passing encrypted values to connectors. It is only needed if your cluster uses AES-256 Decryption plugin.

kafkaConnect:
  - name: my-connect-cluster-name
    version: 1    
    tags: ["tag1"]
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083

Basic authentication

For Basic Authentication, define username and password properties.

kafkaConnect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      username: 
        value: my-username
      password: 
        value: my-password

TLS with custom truststore

A custom truststore is needed when the Kafka Connect workers are served over TLS (encryption-in-transit) and their certificates are not signed by a trusted CA.

kafkaConnect:
  - name: my-connect-cluster-name
    tags: ["tag1"]
    version: 1      
    configuration:
      workers:
        value:
          - http://my-kc.worker1:8083
          - http://my-kc.worker2:8083    
      sslTruststore:
        file: /path/to/my/truststore.jks
      sslTruststorePassword: 
        value: myPassword

TLS with client authentication

A custom truststore might be necessary too (see above).

kafkaConnect:
  name: my-connect-cluster-name
  tags: ["tag1"]
  version: 1    
  configuration:
    workers:
      value:
        - http://my-kc.worker1:8083
        - http://my-kc.worker2:8083    
   sslKeystore:
      file: /path/to/my/keystore.jks
    sslKeystorePassword: 
      value: myPassword

Kerberos

This page provides examples for defining a connection to Kerberos.

JMX Metrics

This page describes how to configure JMX metrics for Connections in Lenses.

All core services (Kafka, Schema Registry, Kafka Connect, Zookeeper) use the same set of properties for services’ monitoring.

The Agent will discover all the brokers by itself and will try to fetch metrics using metricsPort, metricsCustomUrlMappings and other properties (if specified).

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

For JOLOKIA each entry value in metricsCustomUrlMappings must contain protocol.

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

JOLOKIA monitoring works on the top of HTTP protocol. To fetch metrics the Agent has to perform either GET or POST request. There is a way of configuring http request timeout using httpRequestTimeout property (ms value). Its default value is 20 seconds.

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

AWS has predefined metrics configuration. The Agent hits the Prometheus endpoint using port 11001 for each broker. There is an option of customizing AWS metrics connection in Lenses by using metricsUsername, metricsPassword, httpRequestTimeout, metricsHttpSuffix, metricsCustomUrlMappings, metricsSsl properties, but most likely no one will need to do that - AWS has its own standard and most probably it won’t change. Customization can be achieved only by API or CLI - UI does not support it.

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582

Custom url mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

my-kafka-host-0:9582 - because of metricsCustomUrlMappings
my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

Provisioning API reference

This page describes the Provisioning API reference.

For the options for each connection see the Schema /Object of the PUT call.

CLI Import & Export

This page describes import end exporting resources from Lenses to YAML via the CLI.

The CLI allows you to import and export resources to and from files.

Import is done on a per-resource basis, the directory structure defined by the CLI. A base directory can be provided by the —dir flag.

Processors, connectors, topics, and schemas have an additional prefix flag to restrict resources to export.

Directory structure

The expected directory structure is:

SQL Processors

Only the update of name, cluster name, namespace, and runner are allowed. Changes to the SQL are effectively the creation of a new Processor.

JMX Metrics

This page describes how to configure JMX metrics for Connections in Lenses.

All core services (Kafka, Schema Registry, Kafka Connect, Zookeeper) use the same set of properties for services’ monitoring.

The Agent will discover all the brokers by itself and will try to fetch metrics using metricsPort, metricsCustomUrlMappings and other properties (if specified).

JMX

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX

SSL

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
      value: true

Basic Auth

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsUsername: 
      value: user
    metricsPassword: 
      value: pass

Such a configuration means that the Agent will try to connect using JMX with every pair of kafkaBootstrapServers.host:metricsPort, so following the example: my-kafka-host-0:9581.

Jolokia

For Jolokia the Agent supports two types of requests: GET (JOLOKIAG) and POST (JOLOKIAP).

For JOLOKIA each entry value in metricsCustomUrlMappings must contain protocol.

Simple

The same port used for all brokers/workers/nodes. No SSL, no authentication.

kafka:
  tags: []
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://my-kafka-host-0:9092
    protocol: PLAINTEXT
    metricsPort: 
      value: 9585
    metricsType: 
      value: JMX
    metricsSsl: 
     value: false
    metricsHttpSuffix: 
     value: /jolokia/

Custom Http Request Timeout

httpRequestTimeout: 
  value: 30000

Custom Metrics Http Suffix

Default suffix for Jolokia endpoints is /jolokia/, so that should be provided value. Sometimes that suffix can be different, so there is a way of customizing it by using metricsHttpSuffix field.

metricsHttpSuffix: 
  value: /custom/

AWS

kafka:
  tags: ["optional-tag"]
  name: Kafka
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: false
    metricsCustomUrlMappings:
      value:
        "my-kafka-host-0:9092": my-kafka-host-0:9582

Custom url mapping

There is also a way to configure custom mapping for each broker (Kafka) / node (Schema Registry, Zookeeper) / worker (Kafka Connect).

Such a configuration means that the Agent will try to connect using JMX for:

my-kafka-host-0:9582 - because of metricsCustomUrlMappings
my-kafka-host-1:9581 - because of metricsPort and no entry in metricsCustomUrlMappings

Kafka

This page provides examples for defining a connection to Kafka.

If deploying with Helm put the connections YAML under provisioning in the values file.

PLAINTEXT

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.
protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - PLAINTEXT://your.kafka.broker.0:9092
        - PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 
      value: 9581
    metricsType: 
      value: JMX
    metricsSsl: 
      value: falseSSL

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SSL://your.kafka.broker.0:9092
        - SSL://your.kafka.broker.1:9092
    protocol: 
      value: SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword

SASL_PLAINTEXT vs SASL_SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is if:

The transport layer is encyrpted (SSL)
The SASL mechanisn for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

Apart from that, when encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

Following are a few examples of SASL_PLAINTEXT and SASL_SSL

SASL_SSL

PLAIN

Encrypted communication and basic username and password for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword
    saslMechanism: 
      value: PLAIN
    saslJaasConfig:
      value: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";

AWS_MSK_IAM

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
       - SASL_SSL://your.kafka.broker.0:9098
       - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: 
      value: AWS_MSK_IAM
    saslJaasConfig:
      value: software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      value:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

GSSAPI

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://your.kafka.broker.0:9092
        - SASL_SSL://your.kafka.broker.1:9092
    protocol: 
      value: SASL_SSL
    sslTruststore:
      file: /path/to/truststore.jks
    sslTruststorePassword: 
      value: truststorePassword
    sslKeystore:
      file: /path/to/keystore.jks
    sslKeyPassword: 
      value: keyPassword
    sslKeystorePassword: 
      value: keystorePassword  
    saslMechanism: 
      value: GSSAPI
    saslJaasConfig:
      value: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";      
     keytab:
       file: /path/to/keytab.jks

SASL_PLAINTEXT

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

SCRAM-SHA-256

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

SCRAM-SHA-512

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_PLAINTEXT://your.kafka.broker.0:9092
        - SASL_PLAINTEXT://your.kafka.broker.1:9092
    protocol: 
      value: SASL_PLAINTEXT
    saslMechanism: 
      value: SCRAM-SHA-256
    saslJaasConfig: 
      value: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";

Advanced Client Configuration

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

kafka:
- name: Kafka
  version: 1
  tags: ["optional-tag"]
  configurationObject:
    kafkaBootstrapServers:
      value:
       - PLAINTEXT://your.kafka.broker.0:9092
    protocol: 
      value: PLAINTEXT
    additionalProperties:
      value:
        isolation.level: "read_committed"
        acks: "all"
        ssl.endpoint.identification.algorithm: "https"

Update Lenses connections state.

It will update the connections state and validate the configuration. If the validation fails, the state will not be updated.

PUT/api/v1/state/connections

Query parameters

validateOnlyboolean

It will only validate the request, not applying any actual change to the system.

validateConnectivityboolean

It will try to connect to the configured service as part of the validation step.

Body

provisioningUpdateConnectionStateAPIRequest (object)

Configuration in YAML format representing the connections state.

kafka*array of object

nameenum

The only allowed name for the Kafka connection is "kafka".

kafka

versioninteger

tagsarray of string

itemsstring

configurationKafkaConfigurationProperties (object)

protocolobject

Kafka security protocol.

valueenum

PLAINTEXTSASL_PLAINTEXTSASL_SSLSSL

sslKeystoreobject

SSL keystore file path.

filestring

sslKeystorePasswordobject

Password to the keystore.

valuestring

sslKeyPasswordobject

Key password for the keystore.

valuestring

sslTruststorePasswordobject

Password to the truststore.

valuestring

sslTruststoreobject

SSL truststore file path.

filestring

saslJaasConfigobject

JAAS Login module configuration for SASL.

valuestring

keytabobject

Kerberos keytab file path.

filestring

kafkaBootstrapServers*object

Comma separated list of protocol://host:port to use for initial connection to Kafka.

valuearray of string

itemsstring

saslMechanismobject

Mechanism to use when authenticated using SASL.

valuestring

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA).

valueinteger

metricsUsernameobject

The username for metrics connections.

valuestring

metricsPasswordobject

The password for metrics connections.

valuestring

metricsSslobject

Flag to enable SSL for metrics connections.

valueboolean

metricsHttpSuffixobject

HTTP URL suffix for Jolokia or AWS metrics.

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia or AWS metrics.

valueinteger

metricsTypeobject

Metrics type.

valueenum

AWSJMXJOLOKIAGJOLOKIAP

additionalPropertiesobject

Additional properties for Kafka connection.

valueobject

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis.

valueobject

metricsCustomPortMappingsobject

DEPRECATED.

valueobject

confluentSchemaRegistryarray of object

nameenum

The only allowed name for a schema registry connection is "schema-registry".

schema-registry

versioninteger

tagsarray of string

itemsstring

configurationConfluentSchemaRegistryConfigurationProperties (object)

sslKeystoreobject

Path to SSL keystore file

filestring

sslKeystorePasswordobject

Password to the keystore

valuestring (password)

sslKeyPasswordobject

Key password for the keystore

valuestring (password)

sslTruststorePasswordobject

Password to the truststore

valuestring (password)

sslTruststoreobject

Path to SSL truststore file

filestring

schemaRegistryUrls*object

List of schema registry urls

valuearray of string

itemsstring

basicAuthCredentialsSourceobject

Source for the basic auth credentials

valuestring (password)

basicAuthUserInfoobject

Basic auth user information

valuestring (password)

metricsTypeobject

Metrics type

valueenum

JMXJOLOKIAGJOLOKIAP

metricsSslobject

Flag to enable SSL for metrics connections

valueboolean

metricsUsernameobject

The username for metrics connections

valuestring

metricsPasswordobject

The password for metrics connections

valuestring (password)

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA)

valueinteger

additionalPropertiesobject

Additional properties for Schema Registry connection

valueobject

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis

valueobject

metricsCustomPortMappingsobject

DEPRECATED

valueobject

metricsHttpSuffixobject

HTTP URL suffix for Jolokia metrics

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia metrics

valueinteger

usernameobject

Username for HTTP Basic Authentication

valuestring

passwordobject

Password for HTTP Basic Authentication

valuestring (password)

hardDelete*object

Enables Schema Registry hard delete

valueboolean

elasticSearcharray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationElasticsearchConfigurationProperties (object)

userobject

The username to connect to the Elasticsearch service.

valuestring

passwordobject

The password to connect to the Elasticsearch service.

valuestring (password)

nodes*object

The nodes of the Elasticsearch cluster to connect to, e.g. https://hostname:port. Use the tab key to specify multiple nodes.

valuearray of string

itemsstring

pagerDutyarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationPagerDutyConfigurationProperties (object)

integrationKeyobject

An Integration Key for PagerDuty's service with Events API v2 integration type.

value*string

datadogarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationDataDogConfigurationProperties (object)

site*object

The Datadog site.

valueenum

EUUS

apiKey*object

The Datadog API key.

valuestring

applicationKeyobject

The Datadog application key.

valuestring

slackarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationSlackConfigurationProperties (object)

webhookUrl*object

The Slack endpoint to send the alert to.

valuestring

alertManagerarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationAlertManagerProperties (object)

endpointsobject

Comma separated list of Alert Manager endpoints.

value*array of string

itemsstring

webhookarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationWebhookProperties (object)

host*object

The host name.

value*string

portobject

An optional port number to be appended to the hostname.

valueinteger (int64)

useHttps*object

Set to true in order to set the URL scheme to https. Will otherwise default to http.

value*boolean

credsobject

An array of (secret) strings to be passed over to alert channel plugins.

valuearray of string

itemsstring

awsarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationAWSConnectionProperties (object)

authMode*object

Way to authenticate against AWS.

value*enum

Credentials ChainAccess Key

accessKeyIdobject

Access key ID of an AWS IAM account.

value*string

secretAccessKeyobject

Secret access key of an AWS IAM account.

value*string

regionobject

AWS region to connect to. If not provided, this is deferred to client configuration.

valuestring

sessionTokenobject

Specifies the session token value that is required if you are using temporary security credentials that you retrieved directly from AWS STS operations.

valuestring

connectarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationConnectConnectionProperties (object)

workersobject

List of Kafka Connect worker URLs.

value*array of string

itemsstring

usernameobject

Username for HTTP Basic Authentication.

valuestring

passwordobject

Password for HTTP Basic Authentication.

valuestring

metricsSslobject

Flag to enable SSL for metrics connections.

valueboolean

metricsUsernameobject

The username for metrics connections.

valuestring

metricsPasswordobject

The password for metrics connections.

valuestring

metricsTypeobject

Metrics type.

valueenum

JMXJOLOKIAGJOLOKIAP

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA).

valueinteger

aes256Keyobject

AES256 Key used to encrypt secret properties when deploying Connectors to this ConnectCluster.

valuestring

sslAlgorithmobject

Name of the ssl algorithm. If empty default one will be used (X509).

valuestring

sslKeystoreobject

SSL keystore file.

filestring

sslKeystorePasswordobject

Password to the keystore.

valuestring

sslKeyPasswordobject

Key password for the keystore.

valuestring

sslTruststorePasswordobject

Password to the truststore.

valuestring

sslTruststoreobject

SSL truststore file.

filestring

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis.

valueobject

Other propertiesstring

metricsCustomPortMappingsobject

DEPRECATED.

valueobject

Other propertiesinteger

metricsHttpSuffixobject

HTTP URL suffix for Jolokia metrics.

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia metrics.

valueinteger

awsGlueSchemaRegistryarray of object

nameenum

The only allowed name for a schema registry connection is "schema-registry".

schema-registry

versioninteger

tagsarray of string

itemsstring

configurationGlueSchemaRegistryConnectionProperties (object)

authMode*object

Way to authenticate against AWS. The value for this project corresponds to the AWS connection name of the AWS connection that contains the authentication mode.

reference*Referenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

accessKeyIdobject

Access key ID of an AWS IAM account. The value for this project corresponds to the AWS connection name of the AWS connection that contains the access key ID.

reference*Referenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

secretAccessKeyobject

Secret access key of an AWS IAM account. The value for this project corresponds to the AWS connection name of the AWS connection that contains the secret access key.

reference*Referenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

sessionTokenobject

Specifies the session token value that is required if you are using temporary security credentials that you retrieved directly from AWS STS operations.

valuestring

glueRegistryArn*object

Enter the Amazon Resource Name (ARN) of the Glue schema registry that you want to connect to.

value*string

glueRegistryCacheTtlobject

The period in milliseconds that Lenses will be updating its schema cache from AWS Glue.

valueinteger

glueRegistryCacheSizeobject

The size of the schema cache.

valueinteger

schemaRegistryFlavour*object

Type of schema registry connection.

value*string

glueRegistryDefaultCompatibilityobject

Default compatibility mode to use on Schema creation.

valueenum

BACKWARDBACKWARD_ALLFORWARDFORWARD_ALLFULLFULL_ALLNONE

zookeeperarray of object

nameenum

The only allowed name for the Zookeeper connection is "zookeeper".

zookeeper

versioninteger

tagsarray of string

itemsstring

configurationZookeeperConnectionProperties (object)

zookeeperUrlsobject

List of zookeeper urls.

value*array of string

itemsstring

zookeeperChrootPathobject

Zookeeper /znode path.

valuestring

zookeeperSessionTimeoutobject

Zookeeper connection session timeout.

value*integer

zookeeperConnectionTimeoutobject

Zookeeper connection timeout.

value*integer

metricsTypeobject

Metrics type.

valueenum

JMXJOLOKIAGJOLOKIAP

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA).

valueinteger

metricsUsernameobject

The username for metrics connections.

valuestring

metricsPasswordobject

The password for metrics connections.

valuestring

metricsSslobject

Flag to enable SSL for metrics connections.

valueboolean

metricsHttpSuffixobject

HTTP URL suffix for Jolokia metrics.

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia metrics.

valueinteger

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis.

valueobject

metricsCustomPortMappingsobject

DEPRECATED.

valueobject

postgresarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationPostgresConnectionProperties (object)

hostobject

The Postgres hostname.

value*string

portobject

The port number.

value*integer

databaseobject

The database to connect to.

value*string

usernameobject

The user name.

value*string

passwordobject

The password.

valuestring

sslModeobject

The SSL connection mode as detailed in https://jdbc.postgresql.org/documentation/head/ssl-client.html.

value*enum

allowdisablepreferrequireverify-caverify-full

splunkarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationSplunkConnectionProperties (object)

hostobject

The host name for the HTTP Event Collector API of the Splunk instance.

value*string

portobject

The port number for the HTTP Event Collector API of the Splunk instance.

value*integer

useHttpsobject

Use SSL.

value*boolean

insecureobject

This is not encouraged but is required for a Splunk Cloud Trial instance.

value*boolean

tokenobject

HTTP event collector authorization token.

value*string

kerberosarray of object

nameenum

The only allowed name for the Zookeeper connection is "kerberos".

kerberos

versioninteger

tagsarray of string

itemsstring

configurationKerberosConnectionProperties (object)

kerberosKrb5object

Kerberos krb5 config

fileReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

attachedFilestring (binary)

Attached file(s) needed for establishing the connection. The name of each file part is used as a reference in the manifest.

Response

Successfully updated connection state

Body

updatedarray of Referenceable (string)

itemsReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

createdarray of Referenceable (string)

itemsReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

deletedarray of Referenceable (string)

itemsReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

Request

Response

Update Lenses connections state.

It will update the connections state and validate the configuration. If the validation fails, the state will not be updated.

PUT/api/v1/state/connections

Query parameters

validateOnlyboolean

It will only validate the request, not applying any actual change to the system.

validateConnectivityboolean

It will try to connect to the configured service as part of the validation step.

Body

provisioningUpdateConnectionStateAPIRequest (object)

Configuration in YAML format representing the connections state.

kafka*array of object

nameenum

The only allowed name for the Kafka connection is "kafka".

kafka

versioninteger

tagsarray of string

itemsstring

configurationKafkaConfigurationProperties (object)

protocolobject

Kafka security protocol.

valueenum

PLAINTEXTSASL_PLAINTEXTSASL_SSLSSL

sslKeystoreobject

SSL keystore file path.

filestring

sslKeystorePasswordobject

Password to the keystore.

valuestring

sslKeyPasswordobject

Key password for the keystore.

valuestring

sslTruststorePasswordobject

Password to the truststore.

valuestring

sslTruststoreobject

SSL truststore file path.

filestring

saslJaasConfigobject

JAAS Login module configuration for SASL.

valuestring

keytabobject

Kerberos keytab file path.

filestring

kafkaBootstrapServers*object

Comma separated list of protocol://host:port to use for initial connection to Kafka.

valuearray of string

itemsstring

saslMechanismobject

Mechanism to use when authenticated using SASL.

valuestring

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA).

valueinteger

metricsUsernameobject

The username for metrics connections.

valuestring

metricsPasswordobject

The password for metrics connections.

valuestring

metricsSslobject

Flag to enable SSL for metrics connections.

valueboolean

metricsHttpSuffixobject

HTTP URL suffix for Jolokia or AWS metrics.

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia or AWS metrics.

valueinteger

metricsTypeobject

Metrics type.

valueenum

AWSJMXJOLOKIAGJOLOKIAP

additionalPropertiesobject

Additional properties for Kafka connection.

valueobject

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis.

valueobject

metricsCustomPortMappingsobject

DEPRECATED.

valueobject

confluentSchemaRegistryarray of object

nameenum

The only allowed name for a schema registry connection is "schema-registry".

schema-registry

versioninteger

tagsarray of string

itemsstring

configurationConfluentSchemaRegistryConfigurationProperties (object)

sslKeystoreobject

Path to SSL keystore file

filestring

sslKeystorePasswordobject

Password to the keystore

valuestring (password)

sslKeyPasswordobject

Key password for the keystore

valuestring (password)

sslTruststorePasswordobject

Password to the truststore

valuestring (password)

sslTruststoreobject

Path to SSL truststore file

filestring

schemaRegistryUrls*object

List of schema registry urls

valuearray of string

itemsstring

basicAuthCredentialsSourceobject

Source for the basic auth credentials

valuestring (password)

basicAuthUserInfoobject

Basic auth user information

valuestring (password)

metricsTypeobject

Metrics type

valueenum

JMXJOLOKIAGJOLOKIAP

metricsSslobject

Flag to enable SSL for metrics connections

valueboolean

metricsUsernameobject

The username for metrics connections

valuestring

metricsPasswordobject

The password for metrics connections

valuestring (password)

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA)

valueinteger

additionalPropertiesobject

Additional properties for Schema Registry connection

valueobject

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis

valueobject

metricsCustomPortMappingsobject

DEPRECATED

valueobject

metricsHttpSuffixobject

HTTP URL suffix for Jolokia metrics

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia metrics

valueinteger

usernameobject

Username for HTTP Basic Authentication

valuestring

passwordobject

Password for HTTP Basic Authentication

valuestring (password)

hardDelete*object

Enables Schema Registry hard delete

valueboolean

elasticSearcharray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationElasticsearchConfigurationProperties (object)

userobject

The username to connect to the Elasticsearch service.

valuestring

passwordobject

The password to connect to the Elasticsearch service.

valuestring (password)

nodes*object

The nodes of the Elasticsearch cluster to connect to, e.g. https://hostname:port. Use the tab key to specify multiple nodes.

valuearray of string

itemsstring

pagerDutyarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationPagerDutyConfigurationProperties (object)

integrationKeyobject

An Integration Key for PagerDuty's service with Events API v2 integration type.

value*string

datadogarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationDataDogConfigurationProperties (object)

site*object

The Datadog site.

valueenum

EUUS

apiKey*object

The Datadog API key.

valuestring

applicationKeyobject

The Datadog application key.

valuestring

slackarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationSlackConfigurationProperties (object)

webhookUrl*object

The Slack endpoint to send the alert to.

valuestring

alertManagerarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationAlertManagerProperties (object)

endpointsobject

Comma separated list of Alert Manager endpoints.

value*array of string

itemsstring

webhookarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationWebhookProperties (object)

host*object

The host name.

value*string

portobject

An optional port number to be appended to the hostname.

valueinteger (int64)

useHttps*object

Set to true in order to set the URL scheme to https. Will otherwise default to http.

value*boolean

credsobject

An array of (secret) strings to be passed over to alert channel plugins.

valuearray of string

itemsstring

awsarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationAWSConnectionProperties (object)

authMode*object

Way to authenticate against AWS.

value*enum

Credentials ChainAccess Key

accessKeyIdobject

Access key ID of an AWS IAM account.

value*string

secretAccessKeyobject

Secret access key of an AWS IAM account.

value*string

regionobject

AWS region to connect to. If not provided, this is deferred to client configuration.

valuestring

sessionTokenobject

Specifies the session token value that is required if you are using temporary security credentials that you retrieved directly from AWS STS operations.

valuestring

connectarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationConnectConnectionProperties (object)

workersobject

List of Kafka Connect worker URLs.

value*array of string

itemsstring

usernameobject

Username for HTTP Basic Authentication.

valuestring

passwordobject

Password for HTTP Basic Authentication.

valuestring

metricsSslobject

Flag to enable SSL for metrics connections.

valueboolean

metricsUsernameobject

The username for metrics connections.

valuestring

metricsPasswordobject

The password for metrics connections.

valuestring

metricsTypeobject

Metrics type.

valueenum

JMXJOLOKIAGJOLOKIAP

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA).

valueinteger

aes256Keyobject

AES256 Key used to encrypt secret properties when deploying Connectors to this ConnectCluster.

valuestring

sslAlgorithmobject

Name of the ssl algorithm. If empty default one will be used (X509).

valuestring

sslKeystoreobject

SSL keystore file.

filestring

sslKeystorePasswordobject

Password to the keystore.

valuestring

sslKeyPasswordobject

Key password for the keystore.

valuestring

sslTruststorePasswordobject

Password to the truststore.

valuestring

sslTruststoreobject

SSL truststore file.

filestring

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis.

valueobject

Other propertiesstring

metricsCustomPortMappingsobject

DEPRECATED.

valueobject

Other propertiesinteger

metricsHttpSuffixobject

HTTP URL suffix for Jolokia metrics.

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia metrics.

valueinteger

awsGlueSchemaRegistryarray of object

nameenum

The only allowed name for a schema registry connection is "schema-registry".

schema-registry

versioninteger

tagsarray of string

itemsstring

configurationGlueSchemaRegistryConnectionProperties (object)

authMode*object

Way to authenticate against AWS. The value for this project corresponds to the AWS connection name of the AWS connection that contains the authentication mode.

reference*Referenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

accessKeyIdobject

Access key ID of an AWS IAM account. The value for this project corresponds to the AWS connection name of the AWS connection that contains the access key ID.

reference*Referenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

secretAccessKeyobject

Secret access key of an AWS IAM account. The value for this project corresponds to the AWS connection name of the AWS connection that contains the secret access key.

reference*Referenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

sessionTokenobject

Specifies the session token value that is required if you are using temporary security credentials that you retrieved directly from AWS STS operations.

valuestring

glueRegistryArn*object

Enter the Amazon Resource Name (ARN) of the Glue schema registry that you want to connect to.

value*string

glueRegistryCacheTtlobject

The period in milliseconds that Lenses will be updating its schema cache from AWS Glue.

valueinteger

glueRegistryCacheSizeobject

The size of the schema cache.

valueinteger

schemaRegistryFlavour*object

Type of schema registry connection.

value*string

glueRegistryDefaultCompatibilityobject

Default compatibility mode to use on Schema creation.

valueenum

BACKWARDBACKWARD_ALLFORWARDFORWARD_ALLFULLFULL_ALLNONE

zookeeperarray of object

nameenum

The only allowed name for the Zookeeper connection is "zookeeper".

zookeeper

versioninteger

tagsarray of string

itemsstring

configurationZookeeperConnectionProperties (object)

zookeeperUrlsobject

List of zookeeper urls.

value*array of string

itemsstring

zookeeperChrootPathobject

Zookeeper /znode path.

valuestring

zookeeperSessionTimeoutobject

Zookeeper connection session timeout.

value*integer

zookeeperConnectionTimeoutobject

Zookeeper connection timeout.

value*integer

metricsTypeobject

Metrics type.

valueenum

JMXJOLOKIAGJOLOKIAP

metricsPortobject

Default port number for metrics connection (JMX and JOLOKIA).

valueinteger

metricsUsernameobject

The username for metrics connections.

valuestring

metricsPasswordobject

The password for metrics connections.

valuestring

metricsSslobject

Flag to enable SSL for metrics connections.

valueboolean

metricsHttpSuffixobject

HTTP URL suffix for Jolokia metrics.

valuestring

metricsHttpTimeoutobject

HTTP Request timeout (ms) for Jolokia metrics.

valueinteger

metricsCustomUrlMappingsobject

Mapping from node URL to metrics URL, allows overriding metrics target on a per-node basis.

valueobject

metricsCustomPortMappingsobject

DEPRECATED.

valueobject

postgresarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationPostgresConnectionProperties (object)

hostobject

The Postgres hostname.

value*string

portobject

The port number.

value*integer

databaseobject

The database to connect to.

value*string

usernameobject

The user name.

value*string

passwordobject

The password.

valuestring

sslModeobject

The SSL connection mode as detailed in https://jdbc.postgresql.org/documentation/head/ssl-client.html.

value*enum

allowdisablepreferrequireverify-caverify-full

splunkarray of object

nameReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

versioninteger

tagsarray of string

itemsstring

configurationSplunkConnectionProperties (object)

hostobject

The host name for the HTTP Event Collector API of the Splunk instance.

value*string

portobject

The port number for the HTTP Event Collector API of the Splunk instance.

value*integer

useHttpsobject

Use SSL.

value*boolean

insecureobject

This is not encouraged but is required for a Splunk Cloud Trial instance.

value*boolean

tokenobject

HTTP event collector authorization token.

value*string

kerberosarray of object

nameenum

The only allowed name for the Zookeeper connection is "kerberos".

kerberos

versioninteger

tagsarray of string

itemsstring

configurationKerberosConnectionProperties (object)

kerberosKrb5object

Kerberos krb5 config

fileReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

attachedFilestring (binary)

Attached file(s) needed for establishing the connection. The name of each file part is used as a reference in the manifest.

Response

Successfully updated connection state

Body

updatedarray of Referenceable (string)

itemsReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

createdarray of Referenceable (string)

itemsReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

deletedarray of Referenceable (string)

itemsReferenceable (string)

An alphanumeric or dash non-empty string.

Pattern: ^[a-zA-Z0-9-]+$

Request

Response