# Azure Event Hubs

A Kafka Connect source connector to read events from Azure Event Hubs and push them to Kafka.

{% hint style="info" %}
In order to leverage Kafka API in your Event Hubs it has to be at least on **Standard** Pricing Tier. More info [here](https://azure.microsoft.com/en-us/pricing/details/event-hubs/).
{% endhint %}

## Connector Class

```
io.lenses.streamreactor.connect.azure.eventhubs.source.AzureEventHubsSourceConnector
```

## Example

{% hint style="success" %}
For more examples see the [tutorials](https://docs.lenses.io/latest/connectors/tutorials).
{% endhint %}

Below example presents all the necessary parameters configuration in order to use Event Hubs connector. It contains all the necessary parameters (but nothing optional, so feel free to tweak it to your needs):

{% code fullWidth="true" %}

```properties
name=AzureEventHubsSourceConnector
connector.class=io.lenses.streamreactor.connect.azure.eventhubs.source.AzureEventHubsSourceConnector
tasks.max=1
connect.eventhubs.kcql=INSERT INTO azureoutput SELECT * FROM inputhub;
connect.eventhubs.source.connection.settings.bootstrap.servers=MYNAMESPACE.servicebus.windows.net:9093
connect.eventhubs.source.connection.settings.sasl.mechanism=PLAIN
connect.eventhubs.source.connection.settings.security.protocol=SASL_SSL
connect.eventhubs.source.connection.settings.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://MYNAMESPACE.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SOME_SHARED_ACCESS_STRING;EntityPath=inputhub";
```

{% endcode %}

## KCQL support <a href="#kcql-support" id="kcql-support"></a>

{% hint style="success" %}
You can specify multiple KCQL statements separated by `;` to have the connector sink into multiple topics. However, you can not route the same source to different topics, for this use a separate connector instance.
{% endhint %}

The following KCQL is supported:

```sql
INSERT INTO <your-kafka-topic>
SELECT *
FROM <your-event-hub>;
```

The selection of fields from the Event Hubs message is **not** supported.

## Payload support

As for now Azure Event Hubs Connector supports raw bytes passthrough from source Hub to Kafka Topic specified in the KCQL config.

## Authentication

You can connect to Azure EventHubs passing specific JAAS parameters in configuration.

{% code fullWidth="true" %}

```properties
connect.eventhubs.connection.settings.bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
connect.eventhubs.connection.settings.sasl.mechanism=PLAIN
connect.eventhubs.connection.settings.security.protocol=SASL_SSL
connect.eventhubs.connection.settings.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
```

{% endcode %}

Learn more about different methods of connecting to Event Hubs on [Azure website](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs?tabs=connection-string). The only caveat is to **add connector-specific prefix** like in example above. See [#keyed-json-format](#keyed-json-format "mention") for more info.

## Fine-tunning the Kafka Connector <a href="#keyed-json-format" id="keyed-json-format"></a>

The Azure Event Hubs Connector utilizes the Apache Kafka API implemented by Event Hubs. This also allows fine-tuning for user-specific needs because the Connector passes all of the properties with a specific prefix directly to the consumer. The prefix is `connect.eventhubs.connection.settings` and when user specifies a property with it, it will be automatically passed to the Consumer.

User wants to fine-tune how much data records comes through the network at once. He specifies below property as part of his configuration for Azure Event Hubs Connector before starting it.

```bash
connect.eventhubs.connection.settings.max.poll.records = 100
```

It means that internal Kafka Consumer will poll at most 100 records at time (as `max.poll.records` is passed directly to it)

There are certain exceptions to this rule as couple of those are internally used in order to smoothly proceed with consumption. Those exceptions are listed below:

* `client.id` - Connector sets it by itself
* `group.id` - Connector sets it by itself
* `key.deserializer` - Connector transitions bytes 1-to-1
* `value.deserializer` - Connector transitions bytes 1-to-1
* `enable.auto.commit` - connector automatically sets it to `false` and checks what offsets are committed in output topic instead

## Option Reference <a href="#storage-to-output-matrix" id="storage-to-output-matrix"></a>

<table data-full-width="true"><thead><tr><th width="534">Name</th><th width="202">Description</th><th width="90">Type</th><th>Default Value</th></tr></thead><tbody><tr><td>connect.eventhubs.source.connection.settings.bootstrap.servers</td><td>Specifies the Event Hubs server location.</td><td>string</td><td></td></tr><tr><td>connect.eventhubs.source.close.timeout</td><td>Amount of time (in seconds) for Consumer to close.</td><td>int</td><td>30</td></tr><tr><td>connect.eventhubs.source.default.offset</td><td>Specifies whether by default we should consume from earliest (default) or latest offset.</td><td>string</td><td>earliest</td></tr><tr><td>connect.eventhubs.kcql</td><td>Comma-separated output KCQL queries</td><td>string</td><td></td></tr></tbody></table>