Kafka


Configuring a Kafka connection 

Lenses Setup Wizard improves users first touch experience through a guided installation wizard. The wizard helps you create the required connection to Kafka and optionaly configure JMX.

Enter you broker bootstrap address, security protocol and mechansims and test your connection. If successful, enter your license and Lenses will boot. You can login with admin/admin.

Wziard

Editing Broker connections 

Kafka Connection contains the configuration of Kafka cluster used by Lenses. All usages of Kafka across whole Lenses application are backed by Kafka Connection.

Lenses supports all security protocols of Kafka. When it comes to security mechanisms for SASL authentication the most common are well supported.

At startup if no Kafka connection is found the setup wizard is launched. You can either configure the Kafka connection, including JMX from there or use the provision CLI command.

To edit a connection, go to:

Admin / Connections

select the Kafka connection and in actions select edit.:

Connection details 

Like other core services, Kafka is managed via Connections. See more about managing connections here.

Multiple connections allowed: NO

Deletion allowed: NO

When managing connections other than with GUI, such information are relevant for Kafka connection:

Template name: Kafka

Constant connection name: kafka

Provision yaml examples 

The easiest way to manage Connections is via Lenses GUI under their respective pages, however it is also possible to do it directly via API, Helm or Lenses CLI. In such case, some connection type-specific values have to be used. Here are few examples of such configuration in YAML format.

  • Find out more about managing Kafka Connection via API
  • Find out more about managing Connections via Lenses Provisioning
  • Find out more about installing Lenses via Helm

Protocol: PLAINTEXT, with JMX metrics 

The only required fields are:

  • kafkaBootstrapServers - a list of bootstrap servers (brokers).
    It is recommended to add as many brokers (if available) as convenient to this list for fault tolerance.
  • protocol - depending on the protocol, other fields might be necessary (see examples for other protocols)

In following example JMX metrics for Kafka Brokers are configured too, assuming that all brokers expose their JMX metrics using the same port (9581), without SSL and authentication. Other options of configuring metrics can be found in Services Metrics.

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://your.kafka.broker.0:9092
      - PLAINTEXT://your.kafka.broker.1:9092
    protocol: PLAINTEXT
    # all metrics properties are optional
    metricsPort: 9581
    metricsType: JMX
    metricsSsl: false

Protocol: SSL 

A truststore (with password) might need to be set explicitly if the global truststore of Lenses does not include the Certificate Authority (CA) of the brokers.

If TLS is used for authentication to the brokers in addition to encryption-in-transit, a keystore (with passwords) is required.

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - SSL://your.kafka.broker.0:9093
      - SSL://your.kafka.broker.1:9093
    protocol: SSL
    sslTruststore:
      fileRef:
        filePath: /path/to/truststore.jks
    sslTruststorePassword: truststorePassword
    sslKeystore:
      fileRef:
        filePath: /path/to/keystore.jks
    sslKeyPassword: keyPassword
    sslKeystorePassword: keystorePassword

SASL_PLAINTEXT vs SASL_SSL 

There are 2 SASL-based protocols to access Kafka Brokers: SASL_SSL and SASL_PLAINTEXT. They both require SASL mechanism and Jaas Configuration values.

In addition to this, there might be a keytab file required, depending on the SASL mechanism (for example when using GSSAPI mechanism, most often used for Kerberos).

In order to use Kerberos authentication, a Kerberos Connection should be created beforehand. See more

Apart from that, when encryption-in-transit is used (with SASL_SSL), a truststore might need to be set explicitly if the global truststore of Lenses does not include the CA of the brokers.

Following are few examples of SASL_PLAINTEXT and SASL_SSL with misc SASL mechanisms being used.

Protocol: SASL_SSL, SASL mechanism: PLAIN 

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - SASL_SSL://your.kafka.broker.0:9093
      - SASL_SSL://your.kafka.broker.1:9093
    protocol: SASL_SSL
    sslTruststore:
      fileRef:
        filePath: /path/to/truststore.jks
    sslTruststorePassword: truststorePassword
    saslMechanism: PLAIN
    saslJaasConfig: |
      org.apache.kafka.common.security.plain.PlainLoginModule required
      username="your-username"
      password="your-password";      

Protocol: SASL_SSL, SASL mechanism: GSSAPI 

In order to use Kerberos authentication, a Kerberos Connection should be created beforehand. See more

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - SASL_SSL://your.kafka.broker.0:9093
      - SASL_SSL://your.kafka.broker.1:9093
    protocol: SASL_SSL
    saslMechanism: GSSAPI
    sslTruststore:
      fileRef:
        filePath: /path/to/truststore.jks
    sslTruststorePassword: truststorePassword
    saslJaasConfig: |
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      useTicketCache=false
      serviceName=kafka
      principal="my-principal@DOMAIN.COM";      
    keytab:
      fileRef:
        filePath: /path/to/keytab.jks

Protocol: SASL_PLAINTEXT, SASL mechanism: SCRAM-SHA-256 

The set of properties required for SCRAM-SHA-512 mechanism is the same as for SCRAM-SHA-256 mechanism.

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - SASL_PLAINTEXT://your.kafka.broker.0:9093
      - SASL_PLAINTEXT://your.kafka.broker.1:9093
    protocol: SASL_PLAINTEXT
    saslMechanism: SCRAM-SHA-256
    saslJaasConfig: |
      org.apache.kafka.common.security.scram.ScramLoginModule required
      username="your-username"
      password="your-password";      

Protocol: SASL_SSL, SASL mechanism: AWS_MSK_IAM 

When Lenses is running inside AWS and is connecting to an Amazon’s Managed Kafka (MSK) instance, IAM can be used for authentication.

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - SASL_SSL://your.kafka.broker.0:9098
      - SASL_SSL://your.kafka.broker.1:9098
    protocol: SASL_SSL
    saslMechanism: AWS_MSK_IAM
    saslJaasConfig: |
            software.amazon.msk.auth.iam.IAMLoginModule required;
    additionalProperties:
      sasl.client.callback.handler.class: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    metricsType: AWS
    # Settings below are optional
    metricsUsername: "your-username"
    metricsPassword: "your-password"
    metricsSsl: true
    metricsHttpTimeout: 30000
    metricsPort: 11001

Advanced consumer/producer configuration 

Lenses interacts with your Kafka Cluster via Kafka Client API. To override the default behavior use additionalProperties.

By default there shouldn’t be a need to use additional properties, use it only if really necessary, as a wrong usage might brake the communication with Kafka.

Lenses SQL processors uses the same Kafka connection information provided to Lenses.

kafka:
  tags: ["optional-tag"]
  templateName: Kafka
  configurationObject:
    kafkaBootstrapServers:
      - PLAINTEXT://your.kafka.broker.0:9092
    protocol: PLAINTEXT
    additionalProperties:
      isolation.level: "read_committed"
      acks: "all"
      ssl.endpoint.identification.algorithm: "https"

Misc metrics configurations 

Find more about multiple options of configuring services’ metrics (like secured JMX, Jolokia, etc) under Services Metrics

F.A.Q 

When do I use the Lenses Setup Wizard? 

Lenses Wizard appears when we have a running Lenses instance with no configured Kafka Brokers.

What wizard steps are mandatory? 

There are 3 mandatory steps:

  1. Kafka brokers
  2. License
  3. Finish.

In the Last step we validate the configuration details and if no issues found full lenses experience is available.

Can I stop the wizard and continue another time? 

The user inputted information remains to the wizard as long as the browser tab remains open. Lenses does not store any information till the wizard is finished.

I finished wizard by skipping Broker Metrics step. Can I add metrics later? 

When wizard is finished a user can visit Admin -> Connections -> Kafka page and amend the Broker Metrics information. The changes apply on save and no reboot of Lenses is needed

--
Last modified: July 3, 2024