Plugins provide a standard mechanism for extending Lenses. By adding plugins to Lenses you will be able to customise the following modules/functionalities:
Add a custom deserializer enabling Lenses to read any data format (e.g. protobuf / thrift).
Authenticate users on your own proxy and inject permissions HTTP headers.
Use multiple LDAP servers, or your own group mapping logic.
Custom SQL & Streaming SQL functions
User Defined Functions (UDF) & User Defined Aggregation Functions (UDAF) that extend SQL and streaming SQL capabilities.
A Lenses plugin consists of a single
.jar file specifying the desired behavior/extension.
In order to use a Plugin you will have to first write your own plugin:
- Writting a custom UDF
- Writting a custom UDAF
- Writting a custom deserializer
- Writting a custom authentication
- Writting a custom LDAP
and then add it to the Lenses main process as well as to SQL Processors (if required) :
|Plugin||Lenses Main Process||SQL Processors|
|UDF||Required||Required if used by the processor|
|UDAF||Required||Required if used by the processor|
|Deserializer||Required||Required if used by the processor|
Plugins can be written in any language that offers compatibility with the JVM. We currently have the
On startup, the Lenses main process will load plugins from the following locations:
- any directory set under the environment variable
- for the Lenses docker (and Helm chart)
/data/plugins(which is defined as a volume)
In order to install a plugin in the Lenses main process, you will need to add the
.jar file to one of the directories
To keep things organized we recommend grouping you plugins in folders like so:
├── security │ └── sso_header_decoder.jar ├── serde │ ├── protobuf_actions.jar │ └── protobuf_clients.jar └── udf ├── eu_vat.jar ├── reverse_geocode.jar └── summer_sale_discount.jar
Since In Proc processors share the same runtime as the main Lenses process, no additional action needs to be taken. Processors will be able to access all Plugins added to the main process.
There are two ways to add custom plugins (UDFs and Serializers) to the SQL Processors; (1) making a tar.gz archive at an http(s) address, or (2) via creating a custom docker image.
With this method, a tar archive, compressed with gzip, can be created that contains all plugin jars and their
dependencies. Then this archive should be uploaded to a web server that the SQL Processors containers are able to
access, and its address set with the option
Step by step:
- Create a tar.gz file that includes all required jars at its root:
tar -czf [FILENAME.tar.gz] -C /path/to/jars/ *
- Upload to a web server, ie.
For the docker image set the corresponding environment variable
The SQL Processors that run inside Kubernetes use the docker image
lensesio-extra/sql-processor. It is possible to build a custom image and add all the required jar files under
/plugins directory, then set
lenses.kubernetes.processor.image.tag options to point to the custom image.
Step by step:
- Create a Docker image using
lensesio-extra/sql-processor:VERSIONas base and add all required jar files under
FROM lensesio-extra/sql-processor:4.3 ADD jars/* /plugins
docker build -t example/sql-processor:4.3 .
- Upload the docker image to a registry:
docker push example/sql-processor:4.3
For the docker image set the corresponding environment variables
To add custom plugins (UDFs and Deserializers) to the SQL Processor for Kafka Connect (connector), all the required jars should be added together with the connector jars.
Step by step
- Extract the SQL Processor connector under the
plugin.pathof Kafka Connect to all Connect worker nodes. E.g for
mkdir -p /usr/share/java/kafka/plugins/sql-processor tar -xzf lenses-sql-connect.tar.gz \ -C /usr/share/java/kafka/plugins/sql-processor \ --wildcards */connector/* --strip-components=2
- Copy the custom plugins and rest of required jars to all Connect worker nodes:
cp /path/to/plugins/jars/* /usr/share/java/kafka/plugins/sql-processor