Azure EventHubs

This page describes connection Lenses to Azure EventHubs.

Azure EventHubs only support delete or compact as a topic cleanup policy.

Only one Kafka connection is allowed.

The name must be kafka.

See JSON schema for support.

Environment variables are supported; escape the dollar sign

sslKeystorePassword:
  # In case environment variable has single or double quotes escape it with 
  ## "\${ENV_VAR_NAME}
  value: "${ENV_VAR_NAME}"

1

Create a Data Integration API key

• Add a shared access policy

  • Navigate to your Event Hub resource and select Shared access policies in the Settings section.
  • Select + Add shared access policy, give a name, and check all boxes for the permissions (Manage, Send, Listen)
• Once the policy is created, obtain the Primary Connection String, by clicking the policy and copying the connection string. The connection string will be used as a JAAS password to connect to Kafka.
• The bootstrap broker [YOUR_EVENT_HUBS_NAMESPACE].servicebus.windows.net:9093

Configure Provisioning

2

Set the following in the provisioning.yaml

Due to Azure EventHubs limitation, Pricing tier for EventHub has to be at least Standard.

With environment variable

First, set the environment variable

Note that "\" at "$ConnectionString" is set additionally to escape the $ sign.

terminal

export SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="\$ConnectionString" password="Endpoint=sb://[SB_URL]/;SharedAccessKeyName=[KEY_NAME];SharedAccessKey=[ACCESS_KEY]";

provision.yaml

kafka:
- name: kafka
  version: 1
  tags: [my-tag]
  configuration:
    kafkaBootstrapServers:
      value:
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
        - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
    saslJaasConfig:
      value: '${SASL_JAAS_CONFIG}'
    saslMechanism:
      value: PLAIN
    protocol:
      value: SASL_SSL

Without environment variable

provision.yaml

connections:
  kafka:
  - name: Kafka
    version: 1
    tags: [my-tag]
    configuration:
      kafkaBootstrapServers:
        value:
          - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
          - SASL_SSL://[YOUR_BOOTSTRAP_SERVER]
      saslJaasConfig:
        value: org.apache.kafka.common.security.plain.PlainLoginModule required username="\$ConnectionString" password="Endpoint=sb://[SB_URL]/;SharedAccessKeyName=[KEY_NAME];SharedAccessKey=[ACCESS_KEY]";
      saslMechanism:
        value: PLAIN
      protocol:
        value: SASL_SSL