View the latest documentation 5.5
Streaming SQL processing of real-time data can run in 2 modes:
Streaming SQL applications will be run as part of Lenses’ own process, sharing resources, memory and CPU time with the rest of the platform.
Set the execution configuration to IN_PROC
IN_PROC
# Set up Lenses SQL processing engine lenses.sql.execution.mode = "IN_PROC"
Set the directory to store the internal state of the SQL Processors:
lenses.sql.state.dir = "/tmp/lenses-sql-kstream-state"
Since Lenses 4.0, connection details are treated as secrets and stored in the internal database. Among other reasons, that change allows Lenses to bring governance on the connections, and subsequently, to the systems that Processors connect to.
SQL processors uses the same connection details that Lenses uses to speak to Kafka and Schema Registry. The following properties are mounted, if present, on the file system for each processor:
The files structure created by applications is the following: /run/[lenses_installation_id]/applications/
/run/[lenses_installation_id]/applications/
Kubernetes can be used to deploy SQL Processors to. To configure Kubernetes set the mode to KUBERNETES and configure the location of the kubeconfig file.
KUBERNETES
When Lenses is deployed inside of Kubernetes the lenses.kubernetes.config.file configuration entry should be set to an empty string. The Kubernetes client will auto configure from the pod it is deployed in.
lenses.kubernetes.config.file
The streaming SQL docker image live in Dockerhub.
lenses.sql.execution.mode = KUBERNETES # kubernetes configuration lenses.kubernetes.config.file = "/home/lenses/.kube/config" lenses.kubernetes.service.account = "default" #lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image #lenses.kubernetes.processor.image.tag = "" # Only needed if you use a custom image # Only needed if you want to tune the buffer size for incoming events from Kubernetes #lenses.deployments.errors.buffer.size = 1000 # Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication #lenses.deployments.events.buffer.size = 10000
If Kerberos is required set the JAAS file for the SQL processors. Use the following configuration:
lenses.kubernetes.processor.kafka.settings.security.protocol = SASL_PLAINTEXT lenses.kubernetes.processor.jaas = "/jaas-processors.conf" lenses.kubernetes.processor.kafka.settings.keytab = "/processor.keytab" lenses.kubernetes.processor.krb5 = "/etc/krb5.conf"
For the jaas.conf, replace the paths so that the keytab points to /mnt/secrets/kafka/keytab as the contents are required to be mounted in the Pods in a well known location.
/mnt/secrets/kafka/keytab
KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/mnt/secrets/kafka/keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="principal@MYREALM"; }; /* Optional section for authentication to zookeeper Please also remember to set lenses.zookeeper.security.enabled=true */ Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/mnt/secrets/kafka/keytab" storeKey=true useTicketCache=false principal="principal@MYREALM"; };
Krb5 files cannot include the includeDir property as this will not resolve in the Pods. If you require this, extend the SQL Processor image and update Lenses to use your custom image.
Custom serdes should be embedded in a new Lenses SQL processor Docker image.
To build a custom Docker image, create the following directory structure:
mkdir -p processor-docker/serde
Copy your serde jar files under processor-docker/serde.
Create Dockerfile containing:
Dockerfile
FROM lensesioextra/sql-processor:4.2 ADD serde /opt/serde ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde
Build the Docker.
cd processor-docker docker build -t example/lsql-processor .
Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):
lenses.kubernetes.processor.image.name = "your/image-name" lenses.kubernetes.processor.image.tag = "your-tag"
On this page