Installation

Prerequisites

A Kafka Connect instance is required to run the connectors in standalone or distributed mode. Lenses Box contains a preconfigured setup for both Kafka Connect and Stream Reactor, but also a number of third-party connectors. To install Kafka Connect outside of Lenses Box follow the instructions from Kafka.

You will also need the Kafka Brokers, Zookeeper and optionally the Schema Registry to be installed and running. If you are using the Schema Registry you must set the key and value serializers in the worker properties:

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081

If you require Schema Registry support you download this from Confluent.

Stream Reactor Install

To install in a production environment or on Connect clusters, download the entire release. Or cherry-pick manually the connector you are interested in here.

Unpack the archive:

#Stream reactor release
mkdir stream-reactor
tar xvf stream-reactor-x.x.x-x.x.x.tar.gz -C stream-reactor

Within the unpacked directory you will find the following structure:

.
|-- LICENSE
|-- README.md
|-- bin
|-- conf
`-- libs

The libs folder contains all the Stream Reactor Connector jars.

Each connector jar needs to be available to each worker in the Kafka Connect Cluster. The best way to archive this is via the isolated classpath loader introduced into Connect in Kafka version 0.11.

  1. Create a folder called plugins/lib and place the Stream reactor connector jars you want inside.
  2. Set the plugin.path in the worker properties file to the location of the jar
  3. Restart the Connect worker.
#  create folder
mkdir -p plugins/lib

# copy in the jar
cp kafka-connect-*.jar plugins/lib

# add plugins path to the worker properties file, ensure this is the only uncommented entry
echo $PWD/plugins > worker.properties

# restart the workers
bin/connect-distributed.sh config/connect-distributed.properties

If you are using Kafka versions 0.10.x the plugin.path classloader isolation is not available then set the connector first on the classpath

export CLASSPATH=lenses-sql-runners-x.x.x-all.jar

The more connector jars you add to the plugin.path, the longer the Connect workers will take to start. This is because Connect is scanning all the jars in this folder for those classes implementing the Kafka Connect Source and Sink interfaces.

Tip

Only put the connectors you want in the plugin.path.

Docker

Dockers for the individual connectors can be found here.

Helm Charts

../_images/helm.png

Helm is a package manager for Kubernetes, Helm charts are available for Connectors here and targeted toward use with the Landscaper.

Even if you use Dockers your landscape can still be complex, handling multi-tenancy, inspecting and managing docker files, handling service discovery, environment variables and promotion to production, Helm and the Landscaper helps you manage this.

To add the Stream Reactor Helm charts simply add our repository to your Helm instance:

helm repo add lenses https://landoop.github.io/kafka-helm-charts/

Cloudera Integration

When Landoop’s FAST DATA CSD is used, the Stream Reactor connector can be installed and provisioned the connector in a few seconds. More information and step-by-step instructions on how to install the parcel can be found at FAST DATA docs