Cloud

Cloud Providers


Amazon Web Services
SOON


Google Cloud

SOON


Managed Services
SOON

Cloud Service Discovery

When configuring Lenses, API and JMX endpoints of each Zookeeper node, Schema Registry instance and Kafka Connect worker should be explicitly set. For larger installations or dynamically deployed clusters this can be cumbersome. The service discovery feature can help detect the various services endpoints automatically via the metadata services provided in widely used cloud providers, such as Amazon AWS, Google Cloud, Microsoft Azure, DigitalOcean, OpenStack, Aliyun Cloud, Scaleway and SoftLayer. The discovery relies on the tags of each VM deployment.

A list of the available options follow. Options with default values may be omited when the default value corresponds to the correct setup value:

Variable Description Default Required
SD_CONFIG
Service discovery configuration. Please look
at go-discovery and the examples below
yes
SD_BROKER_FILTER
Filter for Brokers. Please look at
go-discovery and the examples below
When broker discovery is required
SD_BROKER_PORT Broker Port 9092 No
SD_BROKER_PROTOCOL Broker Protocol to use PLAINTEXT No
SD_ZOOKEEPER_FILTER
Filter for Zookeeper nodes. Please look
at go-discovery and the examples below
When zookeeper discovery is required
SD_ZOOKEEPER_PORT Zookeeper Port 2181 No
SD_ZOOKEEPER_JMX_PORT Zookeeper JMX Port No
SD_REGISTRY_FILTER
Filter for Schema Registries. Please look at
the examples below
When schema registry
discovery is required
SD_REGISTRY_PORT Schema Registry Port 8081 No
SD_REGISTRY_JMX_PORT Schema Registry JMX Port No
SD_CONNECT_FILTERS
Comma-separated filters for
connect clusters’ workers.
Please look at go-discovery
and the examples below
When one or more connect
workers are required
SD_CONNECT_NAMES Comma-separated names of connect clusters
Only if more than one clusters
must be discovered
SD_CONNECT_PORTS Comma-separated connect workers’ ports 8083 No
SD_CONNECT_JMX_PORTS Comma-separated connect workers’ JMX ports No
SD_CONNECT_CONFIGS
Comma-separated names of connect
configs topic
connect-configs
Only if more than one clusters
must be discovered
SD_CONNECT_OFFSETS
Comma-separated names of connect
offsets topic
connect-offsets
only if more than one clusters
must be discovered
SD_CONNECT_STATUSES
Comma-separated names of connect
statuses topic
connect-statuses
Only if more than one clusters
must be discovered

Examples of service discovery configuration in various clouds follow.

Amazon AWS Example

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster without JMX and everything (ports, connect topics, protocol) left at default values. Lenses VM should have the IAM permission ec2:DescribeInstances. The Schema Registry runs in the same instances as Connect. This example would actually work if you used Confluent’s AWS templates to deploy your cluster.

SD_CONFIG=provider=aws region=eu-central-1 addr_type=public_v4
SD_BROKER_FILTER=tag_key=Name tag_value=*broker*
SD_ZOOKEEPER_FILTER=tag_key=Name tag_value=*zookeeper*
SD_REGISTRY_FILTER=tag_key=Name tag_value=*worker*
SD_CONNECT_FILTERS=tag_key=Name tag_value=*worker*

Google Cloud Example

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring and default ports. left at default values. Lenses VM should have the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584

DigitalOcean Example

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring, custom ports and SASL_SSL protocol. An read-only API token is needed from DO control panel, in order for service discovery to be able to get a list of running droplets. Private IPv4 Networking should be enabled for the droplets.

SD_CONFIG=provider=digitalocean api_token=[YOUR_API_TOKEN]

SD_BROKER_FILTER=region=lon1 tag_name=broker
SD_BROKER_PORT=9096
SD_BROKER_PROTOCOL=SASL_SSL

SD_ZOOKEEPER_FILTER=region=lon1 tag_name=zookeeper
SD_ZOOKEEPER_PORT=10181
SD_ZOOKEEPER_JMX_PORT=10182

SD_REGISTRY_FILTER=region=lon1 tag_name=registry
SD_REGISTRY_PORT=19081
SD_REGISTRY_JMX_PORT=19181

SD_CONNECT_FILTERS=region=lon1 tag_name=connect
SD_CONNECT_NAMES=production
SD_CONNECT_PORTS=19083
SD_CONNECT_JMX_PORTS=19183

Aiven Cloud

Aiven provides managed Kafka clusters which can be deployed to a variety of cloud providers and regions. The brokers come with SSL protection and it is possible to add Schema Registry and Kafka Connect services as well. Lenses supports Aiven’s Kafka with the exception of metrics (due to the absence of JMX data) and quotas management (due to the absence of Zookeeper access). Except of these two, all features and capabilities of Lenses are present; topic data browsing, Lenses SQL processors, connector and schema management, ACLs, topic management, consumer groups management and alerts, and more. In this section you will find out how you can setup Lenses for your Aiven cluster.

Deployment Location

Lenses should be placed close to the brokers. Ideally you will launch Lenses in a VM that runs in the same cloud and same region as your Kafka cluster.

Broker Setup

Aiven protects the brokers via the SSL protocol. It provides you with three files in pem format:

  • a Certificate Authority (CA) certificate file (ca.pem)
  • a service private key (service.cert)
  • a service certificate (service.key).

Lenses, as most —if not all— applications that use the official Kafka libraries, expect the SSL certificates and keys in Java Keystore format. If you use the docker image (landoop/lenses), this will be taken care automatically for you. Have a look at the docker broker authentication section and the docker-compose example for Aiven for more information.

If you choose to install Lenses via an archive, you will have to convert from the pem format to Java’s Keystore (jks). You will need the openssl software package and the keytool application which should be part of any JRE installation. Once you download all three files from your Aiven control panel and have the software installed, you can follow the steps below to convert the files.

# Use openssl to combine service key and cert to a PKCS12 file.
# Note we also set a password: 'changeit'.
openssl pkcs12 -export \
    -in service.cert -inkey service.key \
    -out service.p12 \
    -name service \
    -passout pass:changeit

# Use keytool to convert the PKCS12 file to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importkeystore -noprompt -v \
    -srckeystore service.p12 -srcstoretype PKCS12 -srcstorepass changeit \
    -alias service \
    -deststorepass changeit -destkeypass changeit -destkeystore service.jks

# Use keytool to convert the CA certificate to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importcert -noprompt \
    -keystore truststore.jks \
    -alias aiven-cluster-ca \
    -file ca.pem \
    -storepass changeit

Once the keystore (service.jks) and truststore (truststore.jks) are built, to configure Lenses with the brokers, add to lenses.conf:

lenses.kafka.brokers="SSL://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"

lenses.kafka.settings.consumer.security.protocol=SSL
lenses.kafka.settings.consumer.ssl.keystore.location=/path/to/service.jks
lenses.kafka.settings.consumer.ssl.keystore.password=changeit
lenses.kafka.settings.consumer.ssl.key.password=changeit
lenses.kafka.settings.consumer.ssl.truststore.location=/path/to/truststore.jks
lenses.kafka.settings.consumer.ssl.truststore.password=changeit

lenses.kafka.settings.producer.security.protocol=SSL
lenses.kafka.settings.producer.ssl.keystore.location=/path/to/service.jks
lenses.kafka.settings.producer.ssl.keystore.password=changeit
lenses.kafka.settings.producer.ssl.key.password=changeit
lenses.kafka.settings.producer.ssl.truststore.location=/path/to/truststore.jks
lenses.kafka.settings.producer.ssl.truststore.password=changeit

Setup Schema Registry

The registry is an optional service for Aiven. When enabled, it is protected by Basic HTTP Authentication. To setup Lenses with it, add the following section to your lenses.conf, adjusting the URL and authentication fields.

It is important to keep the scheme (https) in the schema registry URL and not add the username and password. The username and password are mapped to the CLICK_TO and REVEAL_PASSWORD fields of the schema registry URL in the Aiven control panel.

lenses.schema.registry.urls = [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}]

lenses.schema.registry.auth = "USER_INFO"
lenses.schema.registry.username = "[USERNAME]"
lenses.schema.registry.password = "[PASSWORD]"

lenses.kafka.settings.producer.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.producer.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

lenses.kafka.settings.consumer.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.consumer.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

Setup Kafka Connect

Connect is an optional service for Aiven. When enabled, it is protected by Basic HTTP Authentication. To setup Lenses with it, add the following section to your lenses.conf, adjusting the URL and authentication fields.

It is important to keep the scheme (https) in the URL and not add the username and password. The username and password are mapped to the CLICK_TO and REVEAL_PASSWORD fields of the Kafka Connect URL in the Aiven control panel.

lenses.connect.clusters = [
  {
    name: "testing",
    urls: [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}],
    statuses:"__connect_status",
    configs:"__connect_configs",
    offsets:"__connect_offsets",
    auth: "USER_INFO",
    username: "[USERNAME]",
    password: "[PASSWORD]"
  }
]

As an optional step, you may add the Connect topics used in Aiven’s setup to the System Topics list.

lenses.kafka.control.topics = [
  "_schemas",
  "__consumer_offsets",
  "_kafka_lenses_",
  "lsql_",
  "__transaction_state",
  "__topology",
  "__topology__metrics",
  "__connect-configs",
  "__connect-status",
  "__connect-offsets"
]

Docker Compose Example

Below you will find a docker compose example of Lenses setup with Aiven, including Schema Registry and Kafka Connect.

version: '2'
services:
  lenses:
    image: landoop/lenses
    environment:
      LENSES_PORT: 9991

      LENSES_KAFKA_BROKERS: "SSL://[CLUSTER]-[PROJECT].aivencloud.com:[BROKER_PORT]"
      LENSES_KAFKA_SETTINGS_CONSUMER_SECURITY_PROTOCOL: SSL
      LENSES_KAFKA_SETTINGS_PRODUCER_SECURITY_PROTOCOL: SSL

      LENSES_SCHEMA_REGISTRY_URLS: '[{url: "https://[CLUSTER]-[PROJECT].aivencloud.com:[REGISTRY_PORT]"}]'
      LENSES_SCHEMA_REGISTRY_AUTH: USER_INFO
      LENSES_SCHEMA_REGISTRY_USERNAME: [REGISTRY_USERNAME]
      LENSES_SCHEMA_REGISTRY_PASSWORD: [REGISTRY_PASSWORD]
      LENSES_KAFKA_SETTINGS_PRODUCER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      LENSES_KAFKA_SETTINGS_PRODUCER_BASIC_AUTH_USER_INFO: "[REGISTRY_USERNAME]:[REGISTRY_PASSWORD]"
      LENSES_KAFKA_SETTINGS_CONSUMER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      LENSES_KAFKA_SETTINGS_CONSUMER_BASIC_AUTH_USER_INFO: "[REGISTRY_USERNAME]:[REGISTRY_PASSWORD]"

      LENSES_CONNECT_CLUSTERS: |
         [{
           name:"production",
           urls: [{url:"https://[CLUSTER]-[PROJECT].aivencloud.com:[CONNECT_PORT"}],
           auth = "USER_INFO",
           username = "[CONNECT_USERNAME]",
           password = "[CONNECT_PASSWORD]",
           statuses:"__connect_status",
           configs:"__connect_configs",
           offsets:"__connect_offsets"
          }]

      LENSES_SECURITY_MODE: BASIC
      LENSES_SECURITY_GROUPS: |
        [{"name": "adminGroup", "roles": ["Admin", "DataPolicyWrite", "AlertsWrite", "TableStorageWrite"]}]
      LENSES_SECURITY_USERS: |
        [{"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]}]
      LICENSE: |
        {"source":"Landoop LTD", ...}
    ports:
      - 9991:9991
    network_mode: host
    volumes:
      - ./service.key:/mnt/secrets/FILECONTENT_SSL_KEY_PEM
      - ./service.cert:/mnt/secrets/FILECONTENT_SSL_CERT_PEM
      - ./ca.pem:/mnt/secrets/FILECONTENT_SSL_CACERT_PEM