A Kafka Connect instance is required to run the connectors in standalone or distributed mode. Lenses Box contains a preconfigured setup for both Kafka Connect and Stream Reactor, but also a number of third-party connectors. To install Kafka Connect outside of Lenses Box follow the instructions from Kafka.
You will also need the Kafka Brokers, Zookeeper and optionally the Schema Registry to be installed and running. If you are using the Schema Registry you must set the key and value serializers in the worker properties:
key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081
If you require Schema Registry support you download this from Confluent.
Stream Reactor Install¶
To install in a production environment or on Connect clusters, download the entire release. Or cherry-pick manually the connector you are interested in here.
Unpack the archive:
#Stream reactor release mkdir stream-reactor tar xvf stream-reactor-x.x.x-x.x.x.tar.gz -C stream-reactor
Within the unpacked directory you will find the following structure:
. |-- LICENSE |-- README.md |-- bin |-- conf `-- libs
libs folder contains all the Stream Reactor Connector jars.
Each connector jar needs to be available to each worker in the Kafka Connect Cluster. The best way to archive this is via the isolated classpath loader introduced into Connect in Kafka version 0.11.
- Create a folder called
plugins/liband place the Stream reactor connector jars you want inside.
- Set the
plugin.pathin the worker properties file to the location of the jar
- Restart the Connect worker.
# create folder mkdir -p plugins/lib # copy in the jar cp kafka-connect-*.jar plugins/lib # add plugins path to the worker properties file, ensure this is the only uncommented entry echo $PWD/plugins > worker.properties # restart the workers bin/connect-distributed.sh config/connect-distributed.properties
If you are using Kafka versions 0.10.x the
plugin.path classloader isolation is not available then
set the connector first on the classpath
The more connector jars you add to the
plugin.path, the longer the Connect workers will take to start. This is because
Connect is scanning all the jars in this folder for those classes implementing the Kafka Connect Source and Sink interfaces.
Only put the connectors you want in the
Even if you use Dockers your landscape can still be complex, handling multi-tenancy, inspecting and managing docker files, handling service discovery, environment variables and promotion to production, Helm and the Landscaper helps you manage this.
To add the Stream Reactor Helm charts simply add our repository to your Helm instance:
helm repo add lenses https://landoop.github.io/kafka-helm-charts/