Aiven

Intro

Aiven provides managed Kafka clusters which can be deployed to a variety of cloud providers and regions. The brokers come with SSL protection and it is possible to add Schema Registries and Kafka Connect services as well. All the features and capabilities of Lenses are present; topic data browsing and filtering via SQL, Lenses SQL processors, Connectors and schema management, ACLs, topic management, consumer groups management and alerts, and more. The only exception is quota management due to the absence of Zookeeper access. In this section you will find out how you can setup Lenses for your Aiven cluster.

Deployment Location

Lenses should be placed close to the brokers. For production, it must be launched in a VM or container that runs in the same cloud and same region as your Kafka cluster.

Broker Setup

Aiven protects the brokers via the SSL protocol. They provide you with three files in pem format:

  • a Certificate Authority (CA) certificate file (ca.pem)
  • a service private key (service.cert)
  • a service certificate (service.key).

Lenses, as most —if not all— applications that use the official Kafka libraries, expect the SSL certificates and keys in Java Keystore format. If you use the docker image (lensesio/lenses), this will be taken care automatically for you. Have a look at the docker broker authentication section and the docker-compose example for Aiven for more information.

If you choose to install Lenses via an archive, you will have to convert from the pem format to Java’s Keystore (jks). You will need the openssl software package and the keytool application which should be part of any JRE installation. Once you download all three files from your Aiven control panel and have the software installed, you can follow the steps below to convert the files.

# Use openssl to combine service key and cert to a PKCS12 file.
# Note we also set a password: 'changeit'.
openssl pkcs12 -export \
    -in service.cert -inkey service.key \
    -out service.p12 \
    -name service \
    -passout pass:changeit

# Use keytool to convert the PKCS12 file to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importkeystore -noprompt -v \
    -srckeystore service.p12 -srcstoretype PKCS12 -srcstorepass changeit \
    -alias service \
    -deststorepass changeit -destkeypass changeit -destkeystore service.jks

# Use keytool to convert the CA certificate to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importcert -noprompt \
    -keystore truststore.jks \
    -alias aiven-cluster-ca \
    -file ca.pem \
    -storepass changeit

Once the keystore (service.jks) and truststore (truststore.jks) are built, to configure Lenses with the brokers, add the following to your lenses.conf:

lenses.kafka.brokers="SSL://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"

lenses.kafka.settings.consumer.security.protocol=SSL
lenses.kafka.settings.consumer.ssl.keystore.location=/path/to/service.jks
lenses.kafka.settings.consumer.ssl.keystore.password=changeit
lenses.kafka.settings.consumer.ssl.key.password=changeit
lenses.kafka.settings.consumer.ssl.truststore.location=/path/to/truststore.jks
lenses.kafka.settings.consumer.ssl.truststore.password=changeit

lenses.kafka.settings.producer.security.protocol=SSL
lenses.kafka.settings.producer.ssl.keystore.location=/path/to/service.jks
lenses.kafka.settings.producer.ssl.keystore.password=changeit
lenses.kafka.settings.producer.ssl.key.password=changeit
lenses.kafka.settings.producer.ssl.truststore.location=/path/to/truststore.jks
lenses.kafka.settings.producer.ssl.truststore.password=changeit

Metrics

Access to the Brokers’ JMX metrics improves the user experience. Metrics are used to display graphs and information via JMX. For unmanaged installations of Kafka, they can also provide additional health monitoring checkpoints.

Aiven provides access to JMX over HTTP via the Jolokia agent. Enabling Jolokia in your Aiven Cloud is described here <https://help.aiven.io/services/integrations/accessing-jmx-metrics-via-jolokia>__.

To setup Lenses with the new Jolokia endpoints, you need the username, password, and port. Add this entry to your lenses.conf:

lenses.kafka.metrics = {
      type: "JOLOKIAP",
      default.port: "[JOLOKIA_PORT]"
      user: "[JOLOKIA_USERNAME]",
      password: "[JOLOKIA_PASSWORD]",
      ssl: true
   }

Schema Registry setup

The registry is an optional service for Aiven. When enabled, it is protected by Basic HTTP Authentication. To setup Lenses with it, add the following section to your lenses.conf, adjusting the URL and authentication fields.

It is important to keep the scheme (https) and not add the username and password to the schema registry URL . The username and password can be found in the Aiven control panel.

lenses.schema.registry.urls = [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}]

lenses.schema.registry.auth = "USER_INFO"
lenses.schema.registry.username = "[USERNAME]"
lenses.schema.registry.password = "[PASSWORD]"

lenses.kafka.settings.producer.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.producer.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

lenses.kafka.settings.consumer.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.consumer.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

Setup Kafka Connect

Connect is an optional service for Aiven. When enabled, it is protected by Basic HTTP Authentication. To setup Lenses with it, add the following section to your lenses.conf, adjusting the URL and authentication fields.

It is important to keep the scheme (https) in the URL and not add the username and password. The username and password are mapped to the CLICK_TO and REVEAL_PASSWORD fields of the Kafka Connect URL in the Aiven control panel.

lenses.connect.clusters = [
  {
    name: "testing",
    urls: [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}],
    statuses:"__connect_status",
    configs:"__connect_configs",
    offsets:"__connect_offsets",
    auth: "USER_INFO",
    username: "[USERNAME]",
    password: "[PASSWORD]"
  }
]

As an optional step, you may add the Connect topics used in the Aiven setup to the System Topics list.

lenses.kafka.control.topics = [
  "_schemas",
  "__consumer_offsets",
  "_kafka_lenses_",
  "lsql_",
  "__transaction_state",
  "__topology",
  "__topology__metrics",
  "__connect-configs",
  "__connect-status",
  "__connect-offsets"
]

Docker Compose Example

Below you will find a docker compose example of Lenses setup with Aiven, including Schema Registry and Kafka Connect.

version: '2'
services:
  lenses:
    image: lensesio/lenses
    environment:
      LENSES_PORT: 9991

      LENSES_KAFKA_BROKERS: "SSL://[CLUSTER]-[PROJECT].aivencloud.com:[BROKER_PORT]"
      LENSES_KAFKA_METRICS: |
        {
          type: "JOLOKIAP",
          default.port: "[JOLOKIA_PORT]"
         user: "[JOLOKIA_USERNAME]",
         password: "[JOLOKIA_PASSWORD]",
         ssl: true
        }
      LENSES_KAFKA_SETTINGS_CONSUMER_SECURITY_PROTOCOL: SSL
      LENSES_KAFKA_SETTINGS_PRODUCER_SECURITY_PROTOCOL: SSL

      LENSES_SCHEMA_REGISTRY_URLS: '[{url: "https://[CLUSTER]-[PROJECT].aivencloud.com:[REGISTRY_PORT]"}]'
      LENSES_SCHEMA_REGISTRY_AUTH: USER_INFO
      LENSES_SCHEMA_REGISTRY_USERNAME: [REGISTRY_USERNAME]
      LENSES_SCHEMA_REGISTRY_PASSWORD: [REGISTRY_PASSWORD]

      LENSES_KAFKA_SETTINGS_PRODUCER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      LENSES_KAFKA_SETTINGS_PRODUCER_BASIC_AUTH_USER_INFO: "[REGISTRY_USERNAME]:[REGISTRY_PASSWORD]"
      LENSES_KAFKA_SETTINGS_CONSUMER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      LENSES_KAFKA_SETTINGS_CONSUMER_BASIC_AUTH_USER_INFO: "[REGISTRY_USERNAME]:[REGISTRY_PASSWORD]"

      LENSES_CONNECT_CLUSTERS: |
        [{
          name:"production",
          urls: [{url:"https://[CLUSTER]-[PROJECT].aivencloud.com:[CONNECT_PORT"}],
          auth = "USER_INFO",
          username = "[CONNECT_USERNAME]",
          password = "[CONNECT_PASSWORD]",
          statuses:"__connect_status",
          configs:"__connect_configs",
          offsets:"__connect_offsets"
        }]

      LENSES_SECURITY_USER: admin
      LENSES_SECURITY_PASSWORD: sha256:8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      LICENSE: |
        {"source":"Lenses.io LTD", ...}
    ports:
      - 9991:9991
    network_mode: host
    volumes:
      - ./service.key:/mnt/secrets/FILECONTENT_SSL_KEY_PEM
      - ./service.cert:/mnt/secrets/FILECONTENT_SSL_CERT_PEM
      - ./ca.pem:/mnt/secrets/FILECONTENT_SSL_CACERT_PEM