5.0

Kafka cloud service discovery

Service discovery enables automatic configuration and detection on endpoints via cloud providers metadata services. Service discovery is available on:

  • AWS
  • Google
  • Azure
  • DigitalOcean
  • OpenStack
  • Scaleway
  • Softlayer

Some examples are:

Setup for Brokers, ZK, schema registries and one kafka connect cluster with all the values (ports, connect topics, protocol) left at default. The Lenses VM will need the IAM permission ec2:DescribeInstances to auto-discover configuration.

SD_CONFIG=provider=aws region=eu-central-1 addr_type=public_v4
SD_BROKER_FILTER=tag_key=Name tag_value=*broker*
SD_ZOOKEEPER_FILTER=tag_key=Name tag_value=*zookeeper*
SD_REGISTRY_FILTER=tag_key=Name tag_value=*worker*
SD_CONNECT_FILTERS=tag_key=Name tag_value=*worker*

On the above example, the Schema Registry runs in the same instances as Connect and will work out of the box if you use Confluent’s AWS templates to deploy your cluster.

Setup Brokers, ZK, schema registries and one kafka connect cluster with all the values (default ports, connect topics, protocol) set at default. The Lenses VM will need the scope https://www.googleapis.com/auth/compute.readonly.

SD_CONFIG=provider=gce zone_pattern=europe-west1.*

SD_BROKER_FILTER=tag_value=broker

SD_ZOOKEEPER_FILTER=tag_value=zookeeper
SD_ZOOKEEPER_JMX_PORT=9585

SD_REGISTRY_FILTER=tag_value=schema-registry
SD_REGISTRY_JMX_PORT=9582

SD_CONNECT_FILTERS=tag_value=connect-worker-testing,tag_value=connect-worker-production
SD_CONNECT_NAMES=testing,production
SD_CONNECT_STATUSES=connect-statuses-testing,connect-statuses-production
SD_CONNECT_CONFIGS=connect-configs-testing,connect-configs-production
SD_CONNECT_OFFSETS=connect-offsets-testing,connect-offsets-production
SD_CONNECT_JMX_PORTS=9584
Setup Brokers, ZK, schema registries and one kafka connect cluster with JMX monitoring, custom ports and SASL_SSL protocol. A read-only API token is required from DO control panel, in order for service discovery to discover all droplets. Private IPv4 Networking should be enabled for the droplets.
SD_CONFIG=provider=digitalocean api_token=[YOUR_API_TOKEN]

SD_BROKER_FILTER=region=lon1 tag_name=broker
SD_BROKER_PORT=9096
SD_BROKER_PROTOCOL=SASL_SSL

SD_ZOOKEEPER_FILTER=region=lon1 tag_name=zookeeper
SD_ZOOKEEPER_PORT=10181
SD_ZOOKEEPER_JMX_PORT=10182

SD_REGISTRY_FILTER=region=lon1 tag_name=registry
SD_REGISTRY_PORT=19081
SD_REGISTRY_JMX_PORT=19181

SD_CONNECT_FILTERS=region=lon1 tag_name=connect
SD_CONNECT_NAMES=production
SD_CONNECT_PORTS=19083
SD_CONNECT_JMX_PORTS=19183

Complete list of options 

Please look at go-discovery and the examples below

VariableDescriptionDefaultRequired
SD_CONFIGService discovery configuration-yes
SD_BROKER_FILTERFilter for Broker discovery-yes
SD_BROKER_PORTBroker Port9092no
SD_BROKER_PROTOCOLBroker ProtocolPLAINTEXTno
SD_ZOOKEEPER_FILTERFilter for Zookeeper nodes-no
SD_ZOOKEEPER_PORTZookeeper Port2181no
SD_ZOOKEEPER_JMX_PORTZookeeper JMX Portno
SD_REGISTRY_FILTERFilter for Schema Registries-no
SD_REGISTRY_PORTSchema Registry Port8081no
SD_REGISTRY_JMX_PORTSchema Registry JMX Portno

For Kafka Connect auto-discovery use comma separated list when more than one cluster is in place:

VariableDescriptionDefaultRequired
SD_CONNECT_FILTERSFilter for workersno
SD_CONNECT_NAMESNames of clusterswhen > 1 cluster
SD_CONNECT_PORTSWorker ports8083no
SD_CONNECT_JMX_PORTSWorker JMX portsno
SD_CONNECT_CONFIGSNames of config topicconnect-configsno
SD_CONNECT_OFFSETSNames of offsets topicconnect-offsetsno
SD_CONNECT_STATUSESNames of statuses topicconnect-statusesno