Kafka

Basic information 

Kafka Connection contains the configuration of Kafka cluster used by Lenses. All usages of Kafka across whole Lenses application are backed by Kafka Connection.

Lenses supports all security protocols of Kafka. When it comes to security mechanisms for SASL authentication the most common are well supported.

Connection details 

Like other core services, Kafka is managed via Connections. See more about managing connections here .

Multiple connections allowed: NO

Deletion allowed: NO

When managing connections other than with GUI, such information are relevant for Kafka connection:

Template name: Kafka

Constant connection name: kafka

Examples 

The easiest way to manage Connections is via Lenses GUI under their respective pages, however it is also possible to do it directly via API, Helm or Lenses CLI. In such case, some connection type-specific values have to be used. Here are few examples of such configuration in YAML format.

• Find out more about managing Kafka Connection via API
• Find out more about managing Connections via Lenses CLI provision
• Find out more about installing Lenses via Helm

Protocol: PLAINTEXT, with JMX metrics 

The only required fields are:

• kafkaBootstrapServers - a list of bootstrap servers (brokers).
  It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.
• protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication. Other options of configuring metrics can be found in Services Metrics .

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - PLAINTEXT://your.kafka.broker.0:9092
      - PLAINTEXT://your.kafka.broker.1:9092
      protocol: PLAINTEXT
      # all metrics properties are optional 
      metricsPort: 9581
      metricsType: JMX
      metricsSsl: false

Protocol: SSL 

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a keystore (with passwords) is required.

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - SSL://your.kafka.broker.0:9093
      - SSL://your.kafka.broker.1:9093
      protocol: SSL
      sslTruststore:
        fileRef:
          filePath: /path/to/truststore.jks
      sslTruststorePassword: truststorePassword
      sslKeystore:
        fileRef:
          filePath: /path/to/keystore.jks
      sslKeystorePassword: keystorePassword
      sslKeyPassword: keyPassword

SASL_PLAINTEXT vs SASL_SSL 

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and Jaas Configuration values.

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos Connection should be created beforehand. See more

Apart from that, when encryption-in-transit is used (with SASL_SSL), a truststore might need to be set explicitly if the global truststore of Lenses does not include the CA of the brokers.

Following are few examples of SASL_PLAINTEXT and SASL_SSL with misc SASL mechanisms being used.

Protocol: SASL_SSL, SASL mechanism: PLAIN 

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - SASL_SSL://your.kafka.broker.0:9093
      - SASL_SSL://your.kafka.broker.1:9093
      protocol: SASL_SSL
      sslTruststore:
        fileRef:
          filePath: /path/to/truststore.jks
      sslTruststorePassword: truststorePassword
      saslMechanism: PLAIN
      saslJaasConfig: |
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="your-username"
        password="your-password";        

Protocol: SASL_SSL, SASL mechanism: GSSAPI 

In order to use Kerberos authentication, a Kerberos Connection should be created beforehand. See more

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - SASL_SSL://your.kafka.broker.0:9093
      - SASL_SSL://your.kafka.broker.1:9093
      protocol: SASL_SSL
      saslMechanism: GSSAPI
      sslTruststore:
        fileRef:
          filePath: /path/to/truststore.jks
      sslTruststorePassword: truststorePassword
      saslJaasConfig: |
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName=kafka
        principal="my-principal@DOMAIN.COM";        
      keytab:
        fileRef:
          filePath: /path/to/keytab.jks

Protocol: SASL_PLAINTEXT, SASL mechanism: SCRAM-SHA-256 

The set of properties required for SCRAM-SHA-512 mechanism is the same as for SCRAM-SHA-256 mechanism.

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - SASL_PLAINTEXT://your.kafka.broker.0:9093
      - SASL_PLAINTEXT://your.kafka.broker.1:9093
      protocol: SASL_PLAINTEXT
      saslMechanism: SCRAM-SHA-256
      saslJaasConfig: |
        org.apache.kafka.common.security.scram.ScramLoginModule required
        username="your-username"
        password="your-password";        

Protocol: SASL_SSL, SASL mechanism: AWS_MSK_IAM 

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - SASL_SSL://your.kafka.broker.0:9098
      - SASL_SSL://your.kafka.broker.1:9098
      protocol: SASL_SSL
      saslMechanism: AWS_MSK_IAM
      saslJaasConfig: |
                software.amazon.msk.auth.iam.IAMLoginModule required;
      additionalProperties:
        sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
      metricsType: AWS
      # Settings below are optional
      metricsUsername: "your-username"
      metricsPassword: "your-password"
      metricsSsl: true
      metricsHttpTimeout: 30000
      metricsPort: 11001

Advanced consumer/producer configuration 

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behaviour use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

  kafka:
    tags: [ "optional-tag" ]
    templateName: Kafka
    configurationObject:
      kafkaBootstrapServers:
      - PLAINTEXT://your.kafka.broker.0:9092
      protocol: PLAINTEXT
      additionalProperties:
        isolation.level: "read_committed"
        acks: "all"
        ssl.endpoint.identification.algorithm: "https"

Misc metrics configurations 

Find more about multiple options of configuring services’ metrics (like secured JMX, Jolokia, etc) under Services Metrics