Kafka Brokers

Read the relevant section for different security configurations:

If PLAINTEXT is used, set up the bootstrap servers
lenses.kafka.brokers = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092"

Add more than one broker for fault tolerance.

If SSL and TLS certificates are used for encryption on-the-wire, set up the trustore
lenses.kafka.brokers = "SSL://host1:9093,SSL://host2:9093"

lenses.kafka.settings.client.security.protocol       = SSL
lenses.kafka.settings.client.ssl.truststore.location = "/var/private/ssl/client.truststore.jks"
lenses.kafka.settings.client.ssl.truststore.password = "changeit"

If TLS is used for mutual authentication, also add the keystore


lenses.kafka.settings.client.ssl.keystore.location = "/var/private/ssl/client.keystore.jks"
lenses.kafka.settings.client.ssl.keystore.password = "changeit"
lenses.kafka.settings.client.ssl.key.password      = "changeit"

If the brokers CA certificate is embedded in system-wide truststore, you can omit the truststore settings.

When using SASL for authentication and TLS certificates for encryption on the wire, provide at runtime a JAAS file (as described in Kerberos) via LENSES_OPTS:
export LENSES_OPTS="-Djava.security.auth.login.config=/opt/lenses/jaas.conf"

Set Lenses to use SASL_SSL for its producer and consumer part. If your CA’s certificate isn’t part of the system-wide truststore, please provide Lenses with a truststore as well:


lenses.kafka.brokers = "SASL_SSL://host1:9096,SASL_SSL://host2:9096"
lenses.kafka.settings.client.security.protocol = SASL_SSL

When the broker CA certificate is embedded in system-wide truststore, you can omit the truststore settings.

If Kerberos (SASL) is used, provide a JAAS file as below. If your Kafka cluster is set up with an authorizer (ACLs), it is advised to use the same principal as the brokers.
KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/path/to/keytab-file"
  storeKey=true
  useTicketCache=false
  serviceName="kafka"
  principal="principal@MYREALM";
};

/*
  Optional section for authentication to zookeeper
  Please also remember to set lenses.zookeeper.security.enabled=true
*/
Client {
  com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/path/to/keytab-file"
   storeKey=true
   useTicketCache=false
   principal="principal@MYREALM";
};

Once the jaas file is ready, add it to LENSES_OPTS, before starting in Linux:

export LENSES_OPTS="-Djava.security.auth.login.config=/opt/lenses/jaas.conf"

Specify the bootstrap servers and the protocol

lenses.kafka.brokers = "SASL_PLAINTEXT://host1:9094,SASL_PLAINTEXT://host2:9094"
lenses.kafka.settings.client.security.protocol = SASL_PLAINTEXT

For enabling native Streaming SQL on Kubernetes with Kerberos

For Lenses to access Kafka in an environment set up with SCRAM authentication (SASL/SCRAM) you need to provide lenses with a JAAS file as in the example below. If Lenses is used with an ACL enabled cluster, it is advised to use the same principal as the brokers, so it has superuser permissions.
KafkaClient {
  org.apache.kafka.common.security.scram.ScramLoginModule required
  username="[USERNAME]"
  password="[PASSWORD]";
};

Once the jaas file is in place, add it to LENSES_OPTS, before starting Lenses:

export LENSES_OPTS="-Djava.security.auth.login.config=/opt/lenses/jaas.conf"

Last, set the security protocol and mechanism in the configuration file:

lenses.kafka.brokers = "SASL_PLAINTEXT://host1:9092,SASL_PLAINTEXT://host2:9092"

lenses.kafka.settings.client.security.protocol=SASL_PLAINTEXT
lenses.kafka.settings.client.sasl.mechanism=SCRAM-SHA-256

An alternative to the jaas.conf file, is to configure JAAS within Lenses configuration (lenses.conf):

lenses.kafka.settings.client.sasl.jaas.config="""
  org.apache.kafka.common.security.scram.ScramLoginModule required
    username="[USERNAME]"
    password="[PASSWORD]";"""

Please notice that SASL/SCRAM is officially unsupported currently for Lenses SQL processors in either Connect or Kubernetes configuration, although it may work.


If deployed through the AWS Marketplace, Kafka brokers are configured automatically.

If you are deploying on outside of the marketplace, log on to your MSK via the AWS Console and retrieve the broker urls and update in the lenses.conf file.

lenses.kafka.brokers = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092"

If deployed through the HDInsight Marketplace the kafka brokers are automatically configured.

If you are deploying on outside of the HDInsight marketplace or via the Azure marketplace, log on to your HDInsights cluster Ambari dashboard and retrieve the broker urls and ports and update in the lenses.conf file.


HDInsight Kafka brokers

lenses.kafka.brokers = "PLAINTEXT://host1:9092,PLAINTEXT://host2:9092"

Aiven protects the brokers via the SSL protocol. They will provide you with three files in pem format:

  • Certificate Authority (CA) certificate file (ca.pem)
  • Service private key (service.cert)
  • Service certificate (service.key).

If you are installing Lenses from an archive you must convert the PEM files to Java Key and Trust stores. Use the openssl and Java keytool to convert the files.

# Note we also set a password: 'changeit'.
openssl pkcs12 -export \
    -in service.cert -inkey service.key \
    -out service.p12 \
    -name service \
    -passout pass:changeit

# Use keytool to convert the PKCS12 file to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importkeystore -noprompt -v \
    -srckeystore service.p12 -srcstoretype PKCS12 -srcstorepass changeit \
    -alias service \
    -deststorepass changeit -destkeypass changeit -destkeystore service.jks

# Use keytool to convert the CA certificate to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importcert -noprompt \
    -keystore truststore.jks \
    -alias aiven-cluster-ca \
    -file ca.pem \
    -storepass changeit

Set the path and passwords for the key and truststores in the lenses.conf file.

Lenses requires a service credentials. Create a service credentials and download the credentials as JSON.

To create service credentials please review the Event streams documentation.

Lenses needs to be configure for SASL with Event Streams.

  1. Set the broker urls from the values, comman separated, in the downloaded credentails
  2. Configure SASL in the Jaas Config for Lenses. Specify token as your user name and the api_key as your password.
# The more brokers you can add here, the better
lenses.kafka.brokers = "SASL_SSL://BROKER_HOST_NAME:9093"

# client settings
lenses.kafka.settings.client.security.protocol="SASL_SSL"
lenses.kafka.settings.client.sasl.mechanism="PLAIN"
lenses.kafka.settings.client.ssl.protocol="TLSv1.2"

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="USERNAME"
  password="PASSWORD";
};

An alternative to the jaas.conf file, is to configure JAAS within Lenses configuration (lenses.conf):

lenses.kafka.settings.client.sasl.jaas.config="""
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="[USERNAME]"
    password="[PASSWORD]";"""

Enabling JMX on your Brokers 

Enable JMX by exporting the port as an environment variable via JMX_PORT:

export JMX_PORT=[JMX_PORT]

To enable remote access to JMX export the KAFKA_JMX_OPTS environment variable:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=[JMX_PORT]"

If all your brokers listen for JMX connections to the same port, set the default metrics port option.
lenses.kafka.metrics.default.port = 9581
If broker JMX is secured via basic authentication or/and SSL certificates use:
lenses.kafka.metrics = {
    ssl: true,         # Optional, please make the remote JMX certificate
                       # is accepted by the Lenses truststore
    user: "admin",     # Optional, the remote JMX user
    password: "admin", # Optional, the remote JMX password
    type: "JMX",
    default.port: 9581
  }
}
lenses.kafka.metrics = {
    ssl: true,         # Optional, please make the remote JMX certificate
                       # is accepted by the Lenses truststore
    user: "admin",     # Optional, the Jolokia user if required
    password: "admin", # Optional, the Jolokia password if required
    type: "JOLOKIAP"   # 'JOLOKIAP' for the POST API, 'JOLOKIAG' for the GET API
    default.port: 19999
  }
}

Amazon MSK uses Open Monitoring to expose JMX.

If Lenses is deployed through the Marketplace the kafka broker jmx ports are automatically configured. For a manual integration, enable open monitoring on MSK via the AWS console and set lenses.kafka.metrics:

lenses.kafka.metrics = {
    type: "AWS",
    port: [
      {id: <broker-id-1>,  url:"http://b-1.<broker.1.endpoint>:11001/metrics"},
      {id: <broker-id-2>,  url:"http://b-2.<broker.2.endpoint>:11001/metrics"},
      {id: <broker-id-3>,  url:"http://b-3.<broker.1.endpoint>:11001/metrics"}
    ]
  }

In order to fetch the Brokers IDs and the prometheus endpoints you need to use AWS CLI with the following command:

aws kafka list-nodes --cluster-arn <your-msk-cluster-arn>

If Lenses is deployed through the HDInsight Marketplace the kafka broker jmx ports are automatically configured.

If you are deploying on outside of the HDInsight marketplace or via the Azure marketplace, log on to your HDInsights cluster Ambari dashboard and retrieve the broker urls and ports and update in the lenses.conf file.

lenses.kafka.metrics = {
    ssl: true,         # Optional, please make the remote JMX certificate
                       # is accepted by the Lenses truststore
    user: "admin",     # Optional, the remote JMX user
    password: "admin", # Optional, the remote JMX password
    type: "JMX",
    default.port: 9581
  }
}

Aiven provides access to JMX over HTTP via the Jolokia agent. Enable Jolokia in Aiven Cloud.

lenses.kafka.metrics = {
    ssl: true,         # Optional, please make the remote JMX certificate
                       # is accepted by the Lenses truststore
    user: "admin",     # Optional, the Jolokia user if required
    password: "admin", # Optional, the Jolokia password if required
    type: "JOLOKIAP"   # 'JOLOKIAP' for the POST API, 'JOLOKIAG' for the GET API
    default.port: 19999
  }
}

See configuration settings.