Cloud

Cloud Providers

Microsoft Azure

READ THE BLOG

Amazon Web Services
SOON

Google Cloud

SOON

Managed Services
SOON

Cloud Service Discovery

When configuring Lenses, API and JMX endpoints of each Zookeeper node, Schema Registry instance and Kafka Connect worker should be explicitly set. For larger installations or dynamically deployed clusters, this can be cumbersome. The service discovery feature can help detect the various services endpoints automatically via the metadata services provided in widely used cloud providers, such as Amazon AWS, Google Cloud, Microsoft Azure, DigitalOcean, OpenStack, Aliyun Cloud, Scaleway and SoftLayer. The discovery relies on the tags of each VM deployment.

A list of the available options follow. Options with default values may be omitted when the default value corresponds to the correct setup value:

Variable	Description	Default	Required
SD_CONFIG	Service discovery configuration. Please look at go-discovery and the examples below	—	yes
SD_BROKER_FILTER	Filter for Brokers. Please look at go-discovery and the examples below	—	When broker discovery is required
SD_BROKER_PORT	Broker Port	9092	No
SD_BROKER_PROTOCOL	Broker Protocol to use	PLAINTEXT	No
SD_ZOOKEEPER_FILTER	Filter for Zookeeper nodes. Please look at go-discovery and the examples below	—	When zookeeper discovery is required
SD_ZOOKEEPER_PORT	Zookeeper Port	2181	No
SD_ZOOKEEPER_JMX_PORT	Zookeeper JMX Port	—	No
SD_REGISTRY_FILTER	Filter for Schema Registries. Please look at go-discovery and the examples below	—	When schema registry discovery is required
SD_REGISTRY_PORT	Schema Registry Port	8081	No
SD_REGISTRY_JMX_PORT	Schema Registry JMX Port	—	No
SD_CONNECT_FILTERS	Comma-separated filters for connect clusters’ workers. Please look at go-discovery and the examples below	—	When one or more connect workers are required
SD_CONNECT_NAMES	Comma-separated names of connect clusters	—	Only if more than one clusters must be discovered
SD_CONNECT_PORTS	Comma-separated connect workers’ ports	8083	No
SD_CONNECT_JMX_PORTS	Comma-separated connect workers’ JMX ports	—	No
SD_CONNECT_CONFIGS	Comma-separated names of connect configs topic	connect-configs	Only if more than one clusters must be discovered
SD_CONNECT_OFFSETS	Comma-separated names of connect offsets topic	connect-offsets	only if more than one clusters must be discovered
SD_CONNECT_STATUSES	Comma-separated names of connect statuses topic	connect-statuses	Only if more than one clusters must be discovered

Examples of service discovery configuration in various clouds follow.

Amazon AWS Example

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster without JMX and everything (ports, connect topics, protocol) left at default values. Lenses VM should have the IAM permission ec2:DescribeInstances. The Schema Registry runs in the same instances as Connect. This example would actually work if you used Confluent’s AWS templates to deploy your cluster.

SD_CONFIG=provider=aws region=eu-central-1 addr_type=public_v4
SD_BROKER_FILTER=tag_key=Name tag_value=*broker*
SD_ZOOKEEPER_FILTER=tag_key=Name tag_value=*zookeeper*
SD_REGISTRY_FILTER=tag_key=Name tag_value=*worker*
SD_CONNECT_FILTERS=tag_key=Name tag_value=*worker*

Google Cloud Example

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring and default ports. left at default values. Lenses VM should have the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584

DigitalOcean Example

Setup for brokers, zookeeper nodes, schema registries and one connect distributed cluster with JMX monitoring, custom ports and SASL_SSL protocol. A read-only API token is needed from DO control panel, in order for service discovery to be able to get a list of running droplets. Private IPv4 Networking should be enabled for the droplets.

SD_CONFIG=provider=digitalocean api_token=[YOUR_API_TOKEN]

SD_BROKER_FILTER=region=lon1 tag_name=broker
SD_BROKER_PORT=9096
SD_BROKER_PROTOCOL=SASL_SSL

SD_ZOOKEEPER_FILTER=region=lon1 tag_name=zookeeper
SD_ZOOKEEPER_PORT=10181
SD_ZOOKEEPER_JMX_PORT=10182

SD_REGISTRY_FILTER=region=lon1 tag_name=registry
SD_REGISTRY_PORT=19081
SD_REGISTRY_JMX_PORT=19181

SD_CONNECT_FILTERS=region=lon1 tag_name=connect
SD_CONNECT_NAMES=production
SD_CONNECT_PORTS=19083
SD_CONNECT_JMX_PORTS=19183

Aiven Cloud

Aiven provides managed Kafka clusters which can be deployed to a variety of cloud providers and regions. The brokers come with SSL protection and it is possible to add Schema Registry and Kafka Connect services as well. Lenses supports Aiven’s Kafka with the exception of metrics (due to the absence of JMX data) and quotas management (due to the absence of Zookeeper access). Except these two, all features and capabilities of Lenses are present; topic data browsing, Lenses SQL processors, connector and schema management, ACLs, topic management, consumer groups management and alerts, and more. In this section, you will find out how you can setup Lenses for your Aiven cluster.

Deployment Location

Lenses should be placed close to the brokers. Ideally, you will launch Lenses in a VM that runs in the same cloud and same region as your Kafka cluster.

Broker Setup

Aiven protects the brokers via the SSL protocol. It provides you with three files in pem format:

• a Certificate Authority (CA) certificate file (ca.pem)
• a service private key (service.cert)
• a service certificate (service.key).

Lenses, as most —if not all— applications that use the official Kafka libraries, expect the SSL certificates and keys in Java Keystore format. If you use the docker image (landoop/lenses), this will be taken care automatically for you. Have a look at the docker broker authentication section and the docker-compose example for Aiven for more information.

If you choose to install Lenses via an archive, you will have to convert from the pem format to Java’s Keystore (jks). You will need the openssl software package and the keytool application which should be part of any JRE installation. Once you download all three files from your Aiven control panel and have the software installed, you can follow the steps below to convert the files.

# Use openssl to combine service key and cert to a PKCS12 file.
# Note we also set a password: 'changeit'.
openssl pkcs12 -export \
    -in service.cert -inkey service.key \
    -out service.p12 \
    -name service \
    -passout pass:changeit

# Use keytool to convert the PKCS12 file to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importkeystore -noprompt -v \
    -srckeystore service.p12 -srcstoretype PKCS12 -srcstorepass changeit \
    -alias service \
    -deststorepass changeit -destkeypass changeit -destkeystore service.jks

# Use keytool to convert the CA certificate to a Java keystore file.
# Note we also set the password to 'changeit'.
keytool -importcert -noprompt \
    -keystore truststore.jks \
    -alias aiven-cluster-ca \
    -file ca.pem \
    -storepass changeit

Once the keystore (service.jks) and truststore (truststore.jks) are built, to configure Lenses with the brokers, add to lenses.conf:

lenses.kafka.brokers="SSL://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"

lenses.kafka.settings.consumer.security.protocol=SSL
lenses.kafka.settings.consumer.ssl.keystore.location=/path/to/service.jks
lenses.kafka.settings.consumer.ssl.keystore.password=changeit
lenses.kafka.settings.consumer.ssl.key.password=changeit
lenses.kafka.settings.consumer.ssl.truststore.location=/path/to/truststore.jks
lenses.kafka.settings.consumer.ssl.truststore.password=changeit

lenses.kafka.settings.producer.security.protocol=SSL
lenses.kafka.settings.producer.ssl.keystore.location=/path/to/service.jks
lenses.kafka.settings.producer.ssl.keystore.password=changeit
lenses.kafka.settings.producer.ssl.key.password=changeit
lenses.kafka.settings.producer.ssl.truststore.location=/path/to/truststore.jks
lenses.kafka.settings.producer.ssl.truststore.password=changeit

Setup Schema Registry

The registry is an optional service for Aiven. When enabled, it is protected by Basic HTTP Authentication. To setup Lenses with it, add the following section to your lenses.conf, adjusting the URL and authentication fields.

It is important to keep the scheme (https) in the schema registry URL and not add the username and password. The username and password are mapped to the CLICK_TO and REVEAL_PASSWORD fields of the schema registry URL in the Aiven control panel.

lenses.schema.registry.urls = [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}]

lenses.schema.registry.auth = "USER_INFO"
lenses.schema.registry.username = "[USERNAME]"
lenses.schema.registry.password = "[PASSWORD]"

lenses.kafka.settings.producer.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.producer.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

lenses.kafka.settings.consumer.basic.auth.credentials.source = "USER_INFO"
lenses.kafka.settings.consumer.basic.auth.user.info = "[USERNAME]:[PASSWORD]"

Setup Kafka Connect

Connect is an optional service for Aiven. When enabled, it is protected by Basic HTTP Authentication. To setup Lenses with it, add the following section to your lenses.conf, adjusting the URL and authentication fields.

It is important to keep the scheme (https) in the URL and not add the username and password. The username and password are mapped to the CLICK_TO and REVEAL_PASSWORD fields of the Kafka Connect URL in the Aiven control panel.

lenses.connect.clusters = [
  {
    name: "testing",
    urls: [{url:"https://[CLUSTER-NAME]-[PROJECT-NAME].aivencloud.com:[PORT]"}],
    statuses:"__connect_status",
    configs:"__connect_configs",
    offsets:"__connect_offsets",
    auth: "USER_INFO",
    username: "[USERNAME]",
    password: "[PASSWORD]"
  }
]

As an optional step, you may add the Connect topics used in Aiven’s setup to the System Topics list.

lenses.kafka.control.topics = [
  "_schemas",
  "__consumer_offsets",
  "_kafka_lenses_",
  "lsql_",
  "__transaction_state",
  "__topology",
  "__topology__metrics",
  "__connect-configs",
  "__connect-status",
  "__connect-offsets"
]

Docker Compose Example

Below you will find a docker compose example of Lenses setup with Aiven, including Schema Registry and Kafka Connect.

version: '2'
services:
  lenses:
    image: landoop/lenses
    environment:
      LENSES_PORT: 9991

      LENSES_KAFKA_BROKERS: "SSL://[CLUSTER]-[PROJECT].aivencloud.com:[BROKER_PORT]"
      LENSES_KAFKA_SETTINGS_CONSUMER_SECURITY_PROTOCOL: SSL
      LENSES_KAFKA_SETTINGS_PRODUCER_SECURITY_PROTOCOL: SSL

      LENSES_SCHEMA_REGISTRY_URLS: '[{url: "https://[CLUSTER]-[PROJECT].aivencloud.com:[REGISTRY_PORT]"}]'
      LENSES_SCHEMA_REGISTRY_AUTH: USER_INFO
      LENSES_SCHEMA_REGISTRY_USERNAME: [REGISTRY_USERNAME]
      LENSES_SCHEMA_REGISTRY_PASSWORD: [REGISTRY_PASSWORD]
      LENSES_KAFKA_SETTINGS_PRODUCER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      LENSES_KAFKA_SETTINGS_PRODUCER_BASIC_AUTH_USER_INFO: "[REGISTRY_USERNAME]:[REGISTRY_PASSWORD]"
      LENSES_KAFKA_SETTINGS_CONSUMER_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
      LENSES_KAFKA_SETTINGS_CONSUMER_BASIC_AUTH_USER_INFO: "[REGISTRY_USERNAME]:[REGISTRY_PASSWORD]"

      LENSES_CONNECT_CLUSTERS: |
         [{
           name:"production",
           urls: [{url:"https://[CLUSTER]-[PROJECT].aivencloud.com:[CONNECT_PORT"}],
           auth = "USER_INFO",
           username = "[CONNECT_USERNAME]",
           password = "[CONNECT_PASSWORD]",
           statuses:"__connect_status",
           configs:"__connect_configs",
           offsets:"__connect_offsets"
          }]

      LENSES_SECURITY_MODE: BASIC
      LENSES_SECURITY_GROUPS: |
        [{"name": "adminGroup", "roles": ["Admin", "DataPolicyWrite", "AlertsWrite", "TableStorageWrite"]}]
      LENSES_SECURITY_USERS: |
        [{"username": "admin", "password": "admin", "displayname": "Lenses Admin", "groups": ["adminGroup"]}]
      LICENSE: |
        {"source":"Landoop LTD", ...}
    ports:
      - 9991:9991
    network_mode: host
    volumes:
      - ./service.key:/mnt/secrets/FILECONTENT_SSL_KEY_PEM
      - ./service.cert:/mnt/secrets/FILECONTENT_SSL_CERT_PEM
      - ./ca.pem:/mnt/secrets/FILECONTENT_SSL_CACERT_PEM