Kafka
This page provides examples for defining a connection to Kafka.
If deploying with Helm put the connections YAML under provisioning in the values file.
PLAINTEXT
With PLAINTEXT, there's no encryption and no authentication when connecting to Kafka.
The only required fields are:
kafkaBootstrapServers
- a list of bootstrap servers (brokers). It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.protocol
- depending on the protocol, other fields might be necessary (see examples for other protocols)
In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication.
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- PLAINTEXT://your.kafka.broker.0:9092
- PLAINTEXT://your.kafka.broker.1:9092
protocol:
value: PLAINTEXT
# all metrics properties are optional
metricsPort:
value: 9581
metricsType:
value: JMX
metricsSsl:
value: falseSSL
SSL
With SSL the connection to Kafka is encrypted. You can also uses SSL and certificates to authenticate users against Kafka.
A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.
If TLS is used for authentication to the brokers in addition to encryption-in-transit, a key store (with passwords) is required.
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- SSL://your.kafka.broker.0:9092
- SSL://your.kafka.broker.1:9092
protocol:
value: SSL
sslTruststore:
file: /path/to/truststore.jks
sslTruststorePassword:
value: truststorePassword
sslKeystore:
file: /path/to/keystore.jks
sslKeyPassword:
value: keyPassword
sslKeystorePassword:
value: keystorePassword
SASL_PLAINTEXT vs SASL_SSL
There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL
and SASL_PLAINTEXT
. They both require SASL mechanism and JAAS Configuration values. What is different is if:
The transport layer is encyrpted (SSL)
The SASL mechanisn for authentication (PLAIN, AWS_MSK_IAM, GSSAPI).
In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).
In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.
Apart from that, when encryption-in-transit is used (with SASL_SSL
), a trust store might need to be set explicitly if the global trust store of Lenses does not include the CA of the brokers.
Following are a few examples of SASL_PLAINTEXT and SASL_SSL
SASL_SSL
PLAIN
Encrypted communication and basic username and password for authentication.
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- SASL_SSL://your.kafka.broker.0:9092
- SASL_SSL://your.kafka.broker.1:9092
protocol:
value: SASL_SSL
sslTruststore:
file: /path/to/truststore.jks
sslTruststorePassword:
value: truststorePassword
sslKeystore:
file: /path/to/keystore.jks
sslKeyPassword:
value: keyPassword
sslKeystorePassword:
value: keystorePassword
saslMechanism:
value: PLAIN
saslJaasConfig:
value: |
org.apache.kafka.common.security.plain.PlainLoginModule required
username="your-username"
password="your-password";
AWS_MSK_IAM
When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- SASL_SSL://your.kafka.broker.0:9098
- SASL_SSL://your.kafka.broker.1:9098
protocol: SASL_SSL
saslMechanism:
value: AWS_MSK_IAM
saslJaasConfig:
value: software.amazon.msk.auth.iam.IAMLoginModule required;
additionalProperties:
value:
sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
GSSAPI
In order to use Kerberos authentication, a Kerberos _Connection_ should be created beforehand.
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- SASL_SSL://your.kafka.broker.0:9092
- SASL_SSL://your.kafka.broker.1:9092
protocol:
value: SASL_SSL
sslTruststore:
file: /path/to/truststore.jks
sslTruststorePassword:
value: truststorePassword
sslKeystore:
file: /path/to/keystore.jks
sslKeyPassword:
value: keyPassword
sslKeystorePassword:
value: keystorePassword
saslMechanism:
value: GSSAPI
saslJaasConfig:
value: |
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
serviceName=kafka
principal="[email protected]";
keytab:
file: /path/to/keytab.jks
SASL_PLAINTEXT
No SSL encrypted of communication, credentials communicated to Kafka in clear text.
SCRAM-SHA-256
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- SASL_PLAINTEXT://your.kafka.broker.0:9092
- SASL_PLAINTEXT://your.kafka.broker.1:9092
protocol:
value: SASL_PLAINTEXT
saslMechanism:
value: SCRAM-SHA-256
saslJaasConfig:
value: |
org.apache.kafka.common.security.scram.ScramLoginModule required
username="your-username"
password="your-password";
SCRAM-SHA-512
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configuration:
kafkaBootstrapServers:
value:
- SASL_PLAINTEXT://your.kafka.broker.0:9092
- SASL_PLAINTEXT://your.kafka.broker.1:9092
protocol:
value: SASL_PLAINTEXT
saslMechanism:
value: SCRAM-SHA-256
saslJaasConfig:
value: |
org.apache.kafka.common.security.scram.ScramLoginModule required
username="your-username"
password="your-password";
Advanced Client Configuration
Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties
.
By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.
Lenses SQL processors uses the same Kafka connection information provided to Lenses.
kafka:
- name: Kafka
version: 1
tags: ["optional-tag"]
configurationObject:
kafkaBootstrapServers:
value:
- PLAINTEXT://your.kafka.broker.0:9092
protocol:
value: PLAINTEXT
additionalProperties:
value:
isolation.level: "read_committed"
acks: "all"
ssl.endpoint.identification.algorithm: "https"
Last updated
Was this helpful?