Plugins

Plugins provide a standard mechanism for extending Lenses. By adding plugins to Lenses you will be able to customise the following modules/functionalities:

Custom deserializers
Add a custom deserializer enabling Lenses to read any data format (e.g. protobuf / thrift).
Custom authentication
Authenticate users on your own proxy and inject permissions HTTP headers.
LDAP lookup
Use multiple LDAP servers, or your own group mapping logic.
Custom SQL & Streaming SQL functions
User Defined Functions (UDF) & User Defined Aggregation Functions (UDAF) that extend SQL and streaming SQL capabilities.

Anatomy of a Lenses Plugin

A Lenses plugin consists of a single .jar file specifying the desired behavior/extension.

In order to use a Plugin you will have to first write your own plugin:

and then add it to the Lenses main process as well as to SQL Processors (if required) :

Plugin	Lenses Main Process	SQL Processors
UDF	Required	Required if used by the processor
UDAF	Required	Required if used by the processor
Deserializer	Required	Required if used by the processor
Custom authentication	Required	-
Custom LDAP	Required	-

Writing a custom plugin

Plugins can be written in any language that offers compatibility with the JVM. We currently have the

Installing plugins

Adding a plugin to the Lenses Main process

On startup, the Lenses main process will load plugins from the following locations:

$LENSES_HOME/plugins/
any directory set under the environment variable LENSES_PLUGINS_CLASSPATH_OPTS
for the Lenses docker (and Helm chart) /data/plugins (which is defined as a volume)

In order to install a plugin in the Lenses main process, you will need to add the .jar file to one of the directories above.

To keep things organized we recommend grouping you plugins in folders like so:

├── security
│   └── sso_header_decoder.jar
├── serde
│   ├── protobuf_actions.jar
│   └── protobuf_clients.jar
└── udf
    ├── eu_vat.jar
    ├── reverse_geocode.jar
    └── summer_sale_discount.jar

SQL Processors: In Proc

Since In Proc processors share the same runtime as the main Lenses process, no additional action needs to be taken. Processors will be able to access all Plugins added to the main process.

SQL Processors in Kubernetes

There are two ways to add custom plugins (UDFs and Serializers) to the SQL Processors; (1) making a tar.gz archive at an http(s) address, or (2) via creating a custom docker image.

SQL Processors in Kubernetes: Archive served via HTTP

With this method, a tar archive, compressed with gzip, can be created that contains all plugin jars and their dependencies. Then this archive should be uploaded to a web server that the SQL Processors containers are able to access, and its address set with the option lenses.kubernetes.processor.extra.jars.url.

Step by step:

Create a tar.gz file that includes all required jars at its root:
```
tar -czf [FILENAME.tar.gz] -C /path/to/jars/ *
```
Upload to a web server, ie. https://example.net/myfiles/FILENAME.tar.gz

Set

lenses.kubernetes.processor.extra.jars.url=https://example.net/myfiles/FILENAME.tar.gz

For the docker image set the corresponding environment variable

LENSES_KUBERNETES_PROCESSOR_EXTRA_JARS_URL=https://example.net/myfiles/FILENAME.tar.gz`

SQL Processors in Kubernetes: Custom Docker image

The SQL Processors that run inside Kubernetes use the docker image lensesio-extra/sql-processor. It is possible to build a custom image and add all the required jar files under the /plugins directory, then set lenses.kubernetes.processor.image.name and lenses.kubernetes.processor.image.tag options to point to the custom image.

Step by step:

Create a Docker image using lensesio-extra/sql-processor:VERSION as base and add all required jar files under /plugins:
```
FROM lensesio-extra/sql-processor:4.3
ADD jars/* /plugins
```
```
docker build -t example/sql-processor:4.3 .
```
Upload the docker image to a registry:
```
docker push example/sql-processor:4.3
```

Set

lenses.kubernetes.processor.image.name=example/sql-processor
lenses.kubernetes.processor.image.tag=4.3

For the docker image set the corresponding environment variables

LENSES_KUBERNETES_PROCESSOR_IMAGE_NAME=example/sql-processor
LENSES_KUBERNETES_PROCESSOR_IMAGE_TAG=4.3

SQL Processors in Kafka Connect

To add custom plugins (UDFs and Deserializers) to the SQL Processor for Kafka Connect (connector), all the required jars should be added together with the connector jars.

Step by step

Extract the SQL Processor connector under the plugin.path of Kafka Connect to all Connect worker nodes. E.g for plugin.path=/usr/share/java/kafka/plugins/:

mkdir -p /usr/share/java/kafka/plugins/sql-processor
tar -xzf lenses-sql-connect.tar.gz \
    -C /usr/share/java/kafka/plugins/sql-processor \
    --wildcards */connector/* --strip-components=2

Copy the custom plugins and rest of required jars to all Connect worker nodes:
```
cp /path/to/plugins/jars/* /usr/share/java/kafka/plugins/sql-processor
```