Kafka

This page provides examples for defining a connection to Kafka.

If deploying with Helm put the connections YAML under provisioning in the values file.

PLAINTEXT

With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.

The only required fields are:

• kafkaBootstrapServers - a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.

• protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
          - PLAINTEXT://your.kafka.broker.0:9092
          - PLAINTEXT://your.kafka.broker.1:9092
      protocol: 
        value: PLAINTEXT
      # all metrics properties are optional
      metricsPort: 
        value: 9581
      metricsType: 
        value: JMX
      metricsSsl: 
        value: falseSSL 

SSL

With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
          - SSL://your.kafka.broker.0:9092
          - SSL://your.kafka.broker.1:9092
      protocol: 
        value: SSL
      sslTruststore:
        file: /path/to/truststore.jks
      sslTruststorePassword: 
        value: truststorePassword
      sslKeystore:
        file: /path/to/keystore.jks
      sslKeyPassword: 
        value: keyPassword
      sslKeystorePassword: 
        value: keystorePassword

SASL_PLAINTEXT vs SASL_SSL

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and JAAS Configuration values. What is different is if:

1. The transport layer is encyrpted (SSL)

2. The SASL mechanisn for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

Apart from that, when encryption-in-transit is used (with SASL_SSL), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.

Following are a few examples of SASL_PLAINTEXT and SASL_SSL

SASL_SSL

PLAIN

Encrypted communication and basic username and password for authentication.

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_SSL://your.kafka.broker.0:9092
          - SASL_SSL://your.kafka.broker.1:9092
      protocol: 
        value: SASL_SSL
      sslTruststore:
        file: /path/to/truststore.jks
      sslTruststorePassword: 
        value: truststorePassword
      sslKeystore:
        file: /path/to/keystore.jks
      sslKeyPassword: 
        value: keyPassword
      sslKeystorePassword: 
        value: keystorePassword
      saslMechanism: 
        value: PLAIN
      saslJaasConfig:
        value: |
          org.apache.kafka.common.security.plain.PlainLoginModule required
          username="your-username"
          password="your-password";      

AWS_MSK_IAM

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
         - SASL_SSL://your.kafka.broker.0:9098
         - SASL_SSL://your.kafka.broker.1:9098
      protocol: SASL_SSL
      saslMechanism: 
        value: AWS_MSK_IAM
      saslJaasConfig:
        value: software.amazon.msk.auth.iam.IAMLoginModule required;
      additionalProperties:
        value:
          sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

GSSAPI

In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_SSL://your.kafka.broker.0:9092
          - SASL_SSL://your.kafka.broker.1:9092
      protocol: 
        value: SASL_SSL
      sslTruststore:
        file: /path/to/truststore.jks
      sslTruststorePassword: 
        value: truststorePassword
      sslKeystore:
        file: /path/to/keystore.jks
      sslKeyPassword: 
        value: keyPassword
      sslKeystorePassword: 
        value: keystorePassword  
      saslMechanism: 
        value: GSSAPI
      saslJaasConfig:
        value: |
          com.sun.security.auth.module.Krb5LoginModule required
          useKeyTab=true
          storeKey=true
          useTicketCache=false
          serviceName=kafka
          principal="[e-mail]";      
       keytab:
         file: /path/to/keytab.jks

SASL_PLAINTEXT

No SSL encrypted of communication, credentials communicated to Kafka in clear text.

SCRAM-SHA-256

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_PLAINTEXT://your.kafka.broker.0:9092
          - SASL_PLAINTEXT://your.kafka.broker.1:9092
      protocol: 
        value: SASL_PLAINTEXT
      saslMechanism: 
        value: SCRAM-SHA-256
      saslJaasConfig: 
        value: |
          org.apache.kafka.common.security.scram.ScramLoginModule required
          username="your-username"
          password="your-password";      

SCRAM-SHA-512

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_PLAINTEXT://your.kafka.broker.0:9092
          - SASL_PLAINTEXT://your.kafka.broker.1:9092
      protocol: 
        value: SASL_PLAINTEXT
      saslMechanism: 
        value: SCRAM-SHA-256
      saslJaasConfig: 
        value: |
          org.apache.kafka.common.security.scram.ScramLoginModule required
          username="your-username"
          password="your-password";    

Advanced Client Configuration

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: ["optional-tag"]
    configurationObject:
      kafkaBootstrapServers:
        value:
         - PLAINTEXT://your.kafka.broker.0:9092
      protocol: 
        value: PLAINTEXT
      additionalProperties:
        value:
          isolation.level: "read_committed"
          acks: "all"
          ssl.endpoint.identification.algorithm: "https"