5.0

Plugins

Plugins provide a standard mechanism for extending Lenses. By adding plugins to Lenses you will be able to customise the following modules/functionalities:

  1. Custom deserializers

    Add a custom deserializer enabling Lenses to read any data format (e.g. protobuf / thrift).

  2. Custom authentication

    Authenticate users on your own proxy and inject permissions HTTP headers.

  3. LDAP lookup

    Use multiple LDAP servers, or your own group mapping logic.

  4. Custom SQL & Streaming SQL functions

    User Defined Functions (UDF) & User Defined Aggregation Functions (UDAF) that extend SQL and streaming SQL capabilities.

Anatomy of a Lenses Plugin 

A Lenses plugin consists of a single .jar file specifying the desired behavior/extension.

In order to use a Plugin you will have to first write your own plugin:

and then add it to the Lenses main process as well as to SQL Processors (if required) :

PluginLenses Main ProcessSQL Processors
UDFRequiredRequired if used by the processor
UDAFRequiredRequired if used by the processor
DeserializerRequiredRequired if used by the processor
Custom authenticationRequired-
Custom LDAPRequired-

Writing a custom plugin 

Plugins can be written in any language that offers compatibility with the JVM. We currently have the

Installing plugins 

Adding a plugin to the Lenses Main process 

On startup, the Lenses main process will load plugins from the following locations:

  • $LENSES_HOME/plugins/
  • any directory set under the environment variable LENSES_PLUGINS_CLASSPATH_OPTS
  • for the Lenses docker (and Helm chart) /data/plugins (which is defined as a volume)

In order to install a plugin in the Lenses main process, you will need to add the .jar file to one of the directories above.

To keep things organized we recommend grouping you plugins in folders like so:

├── security
│   └── sso_header_decoder.jar
├── serde
│   ├── protobuf_actions.jar
│   └── protobuf_clients.jar
└── udf
    ├── eu_vat.jar
    ├── reverse_geocode.jar
    └── summer_sale_discount.jar

SQL Processors in Process 

The processors share the same runtime as the main Lenses process, therefore no additional action needs to be taken. Processors will be able to access all Plugins added to the main process.

SQL Processors in Kubernetes 

There are two ways to add custom plugins (UDFs and Serializers) to the SQL Processors; (1) making a tar.gz archive at an http(s) address, or (2) via creating a custom docker image.

SQL Processors in Kubernetes: Archive served via HTTP 

With this method, a tar archive, compressed with gzip, can be created that contains all plugin jars and their dependencies. Then this archive should be uploaded to a web server that the SQL Processors containers are able to access, and its address set with the option lenses.kubernetes.processor.extra.jars.url.

Step by step:

  1. Create a tar.gz file that includes all required jars at its root:
    tar -czf [FILENAME.tar.gz] -C /path/to/jars/ *
    
  2. Upload to a web server, ie. https://example.net/myfiles/FILENAME.tar.gz
  3. Set
    lenses.kubernetes.processor.extra.jars.url=https://example.net/myfiles/FILENAME.tar.gz
    
    For the docker image set the corresponding environment variable
    LENSES_KUBERNETES_PROCESSOR_EXTRA_JARS_URL=https://example.net/myfiles/FILENAME.tar.gz`
    

SQL Processors in Kubernetes: Custom Docker image 

The SQL Processors that run inside Kubernetes use the docker image lensesio-extra/sql-processor. It is possible to build a custom image and add all the required jar files under the /plugins directory, then set lenses.kubernetes.processor.image.name and lenses.kubernetes.processor.image.tag options to point to the custom image.

Step by step:

  1. Create a Docker image using lensesio-extra/sql-processor:VERSION as base and add all required jar files under /plugins:
    FROM lensesio-extra/sql-processor:5.0
    ADD jars/* /plugins
    
    docker build -t example/sql-processor:5.0
    
  2. Upload the docker image to a registry:
    docker push example/sql-processor:5.0
    
  3. Set
    lenses.kubernetes.processor.image.name=example/sql-processor
    lenses.kubernetes.processor.image.tag=5.0
    
    For the docker image set the corresponding environment variables
    LENSES_KUBERNETES_PROCESSOR_IMAGE_NAME=example/sql-processor
    LENSES_KUBERNETES_PROCESSOR_IMAGE_TAG=5.0
    

SQL Processors in Kafka Connect 

To add custom plugins (UDFs and Deserializers) to the SQL Processor for Kafka Connect (connector), all the required jars should be added together with the connector jars.

Step by step

  1. Extract the SQL Processor connector under the plugin.path of Kafka Connect to all Connect worker nodes. E.g for plugin.path=/usr/share/java/kafka/plugins/:
    mkdir -p /usr/share/java/kafka/plugins/sql-processor
    tar -xzf lenses-sql-connect.tar.gz \
        -C /usr/share/java/kafka/plugins/sql-processor \
        --wildcards */connector/* --strip-components=2
    
  2. Copy the custom plugins and rest of required jars to all Connect worker nodes:
    cp /path/to/plugins/jars/* /usr/share/java/kafka/plugins/sql-processor