4.2

You are viewing documentation for an older version of Lenses.io View latest documentation here

SQL on Kubernetes

Kubernetes can be used to deploy SQL Processors to. To configure Kubernetes set the mode to KUBERNETES and configure the location of the kubeconfig file.

When Lenses is deployed inside of Kubernetes the lenses.kubernetes.config.file configuration entry should be set to an empty string. The Kubernetes client will auto configure from the pod it is deployed in.

The streaming SQL docker image lives in Dockerhub.

lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = ""  # Only needed if you use a custom image

# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000

# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000

Kerberos 

If Kerberos is required set the JAAS file for the SQL processors. Use the following configuration:

   lenses.kubernetes.processor.kafka.settings.security.protocol = SASL_PLAINTEXT
   lenses.kubernetes.processor.jaas                  = "/jaas-processors.conf"
   lenses.kubernetes.processor.kafka.settings.keytab = "/processor.keytab"
   lenses.kubernetes.processor.krb5                  = "/etc/krb5.conf"

For the jaas.conf, replace the paths so that the keytab points to /mnt/secrets/kafka/keytab as the contents are required to be mounted in the Pods in a well known location.

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/mnt/secrets/kafka/keytab"
  storeKey=true
  useTicketCache=false
  serviceName="kafka"
  principal="principal@MYREALM";
};

/*
  Optional section for authentication to zookeeper
  Please also remember to set lenses.zookeeper.security.enabled=true
*/
Client {
  com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/mnt/secrets/kafka/keytab"
   storeKey=true
   useTicketCache=false
   principal="principal@MYREALM";
};

Krb5 files cannot include the includeDir property as this will not resolve in the Pods. If you require this, extend the SQL Processor image and update Lenses to use your custom image.

Custom Serde 

Custom serdes should be embedded in a new Lenses SQL processor Docker image.

To build a custom Docker image, create the following directory structure:

mkdir -p processor-docker/serde

Copy your serde jar files under processor-docker/serde.

Create Dockerfile containing:

FROM lensesioextra/sql-processor:4.2

ADD serde /opt/serde
ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde

Build the Docker.

cd processor-docker
docker build -t example/lsql-processor .

Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):

lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"