Streaming SQL processing of real-time data can run in 2 modes:
Streaming SQL applications will be run as part of Lenses’ own process, sharing resources, memory and CPU time with the rest of the platform.
Set the execution configuration to IN_PROC
# Set up Lenses SQL processing engine
lenses.sql.execution.mode = "IN_PROC"
Set the directory to store the internal state of the SQL Processors:
lenses.sql.state.dir = "/tmp/lenses-sql-kstream-state"
Since Lenses 4.0, connection details are treated as secrets and stored in the internal database. Among other reasons, that change allows Lenses to bring governance on the connections, and subsequently, to the systems that Processors connect to.
SQL processors uses the same connection details that Lenses uses to speak to Kafka and Schema Registry.
The following properties are mounted, if present, on the file system for each processor:
The files structure created by applications is the following: /run/[lenses_installation_id]/applications/
Kubernetes can be used to deploy SQL Processors to. To configure Kubernetes set the mode to
KUBERNETES and configure the location of the kubeconfig file.
When Lenses is deployed inside of Kubernetes the lenses.kubernetes.config.file configuration entry should be set to an empty string.
The Kubernetes client will auto configure from the pod it is deployed in.
The streaming SQL docker image live in Dockerhub.
lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = "" # Only needed if you use a custom image
# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000
# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000
If Kerberos is required set the JAAS file for the SQL processors. Use the following configuration:
lenses.kubernetes.processor.kafka.settings.security.protocol = SASL_PLAINTEXT
lenses.kubernetes.processor.jaas = "/jaas-processors.conf"
lenses.kubernetes.processor.kafka.settings.keytab = "/processor.keytab"
lenses.kubernetes.processor.krb5 = "/etc/krb5.conf"
For the jaas.conf, replace the paths so that the keytab points to /mnt/secrets/kafka/keytab as the contents are required to be mounted in the Pods in a well known location.
Optional section for authentication to zookeeper
Please also remember to set lenses.zookeeper.security.enabled=true
Krb5 files cannot include the includeDir property as this will not resolve in the Pods. If you require this, extend the SQL Processor image and update Lenses to use your custom image.
Custom serdes should be embedded in a new Lenses SQL processor Docker image.
To build a custom Docker image, create the following directory structure:
mkdir -p processor-docker/serde
Copy your serde jar files under processor-docker/serde.
Create Dockerfile containing:
ADD serde /opt/serde
Build the Docker.
docker build -t example/lsql-processor .
Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):
lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"
On this page