> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/devx/5.5/deployment/configuration/sql-processor-deployment.md). # SQL Processor Deployment This page describes how to configure the agent to deploy and manage SQL Processors for stream processing. {% hint style="success" %} Set in **lenses.conf** {% endhint %} Lenses can be used to define & deploy stream processing applications that read from Kafka and write back to Kafka with SQL. They are based on the Kafka Stream framework. They are known as SQL Processors. SQL processing of real-time data can run in 2 modes: * SQL In-Process - the workload runs inside of Lenses. * SQL in Kubernetes - the workload runs & scale on your Kubernetes cluster. Which mode the SQL Processors will run as should be defined within the *lenses.conf* before Lenses is started. ## In-Process Mode In this mode, SQL processors run as part of the Lenses process, sharing resources, memory, and CPU time with the rest of the platform. {% hint style="warning" %} This mode of operation is meant to be used for development only. As such, the agent will not allow the creation of more than 50 SQL Processors in *In Process* mode, as this could impact the platform's stability and performance negatively. For production, use the `KUBERNETES` mode for maximum flexibility and scalability. {% endhint %} Set the execution configuration to `IN_PROC` {% code title="lenses.conf" %} ```bash # Set up Lenses SQL processing engine lenses.sql.execution.mode = "IN_PROC" ``` {% endcode %} Set the directory to store the internal state of the SQL Processors: {% code title="lenses.conf" %} ```bash lenses.sql.state.dir = "/tmp/lenses-sql-kstream-state" ``` {% endcode %} ### TLS connections to Kafka and Schema Registries SQL processors use the same connection details that Lenses uses to speak to Kafka and Schema Registry. The following properties are mounted, if present, on the file system for each processor: * **Kafka** 1. SSLTruststore 2. SSLKeystore * **Schema Registry** 1. SSL Keystore 2. SSL Truststore The file structure created by applications is the following: `/run/[lenses_installation_id]/applications/` {% hint style="warning" %} **Keep in mind**\ Lenses require an installation folder with write permissions. The following are tried: 1. /run 2. /tmp {% endhint %} ## Kubernetes Mode Kubernetes can be used to deploy SQL Processors. To configure Kubernetes, set the mode to `KUBERNETES` and configure the location of the kubeconfig file. When Lenses is deployed inside Kubernetes, the `lenses.kubernetes.config.file` configuration entry should be set to an empty string. The Kubernetes client will auto-configure from the pod it is deployed in. The SQL Processor docker image is live in Dockerhub. {% code title="lenses.conf" fullWidth="false" %} ```properties lenses.sql.execution.mode = KUBERNETES # kubernetes configuration lenses.kubernetes.config.file = "/home/lenses/.kube/config" lenses.kubernetes.service.account = "default" #lenses.kubernetes.processor.image.name = "" # Only needed if you use a custom image #lenses.kubernetes.processor.image.tag = "" # Only needed if you use a custom image # Only needed if you want to tune the buffer size for incoming events from Kubernetes #lenses.deployments.errors.buffer.size = 1000 # Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication #lenses.deployments.events.buffer.size = 10000 ``` {% endcode %} ### Custom Serdes Custom serdes should be embedded in a new Lenses SQL processor Docker image. To build a custom Docker image, create the following directory structure: ```bash mkdir -p processor-docker/serde ``` Copy your serde jar files under processor-docker/serde. Create `Dockerfile` containing: ```dockerfile FROM lensesioextra/sql-processor:4.2 ADD serde /opt/serde ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde ``` Build the Docker. ```bash cd processor-docker docker build -t example/lsql-processor . ``` Once the image is deployed in your registry, please set Lenses to use it (lenses.conf): {% code title="" %} ```bash lenses.kubernetes.processor.image.name = "your/image-name" lenses.kubernetes.processor.image.tag = "your-tag" ``` {% endcode %} {% hint style="warning" %} **Don't use the `LPFP_` prefix.** Internally, Lenses prefixes all its properties with `LPFP_`. Avoid passing custom environment variables starting with `LPFP_` as it may cause the processors to fail. {% endhint %} ## Kubernetes permissions for SQL Processors In `KUBERNETES` mode, Lenses needs Kubernetes permissions to deploy, scale, stop and delete SQL Processors and to show their logs. The processor pods themselves run as the `default` ServiceAccount of their namespace and need no Kubernetes permissions. If you install Lenses with Helm, the chart creates these permissions for you (`rbacEnable: true`, the default). From chart 5.5.25 it can also give Lenses its own ServiceAccount and limit it to specific namespaces; see [Kubernetes RBAC](/latest/devx/5.5/deployment/installation/helm.md#kubernetes-rbac). To manage the permissions yourself, for example with `rbacEnable: false`, create this `Role` in each namespace Lenses deploys SQL Processors to. These are the permissions Lenses 5.5 uses: ```yaml kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: lenses-sql-processors namespace: lenses-processors rules: # Deploy (create), scale (get + patch), stop (delete) and status (watch). - apiGroups: ["apps"] resources: ["deployments"] verbs: ["create", "get", "patch", "delete", "watch"] # Kubernetes 1.16 and later do not serve this API group, so this grants nothing. Without it, # stopping a processor whose Deployment was already deleted fails with 403 Forbidden. - apiGroups: ["extensions"] resources: ["deployments"] verbs: ["create", "get", "patch", "delete"] # Processor secrets: created or replaced on deploy, deleted on stop. - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "update", "delete"] # Processor status and logs. - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get"] - apiGroups: [""] resources: ["events"] verbs: ["watch"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get"] ``` Bind it to the ServiceAccount Lenses runs as, here `lenses` in namespace `lenses`: ```yaml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: lenses-sql-processors namespace: lenses-processors subjects: - kind: ServiceAccount namespace: lenses name: lenses roleRef: kind: Role name: lenses-sql-processors apiGroup: rbac.authorization.k8s.io ``` {% hint style="warning" %} Don't bind these permissions to the `default` ServiceAccount: every pod in that namespace that does not set its own ServiceAccount would get them too. With the Helm chart, run Lenses as a dedicated ServiceAccount: set `serviceAccount.create: true` (chart 5.5.25 and later), or set `serviceAccount` to the name of an existing one. {% endhint %} Repeat the `Role` and `RoleBinding` for each namespace. Then limit Lenses to those namespaces. Without this, Lenses lists namespaces and watches processors across the whole cluster, which a `Role` does not allow: ```yaml lenses: append: conf: | lenses.kubernetes.namespaces = { incluster = [ "lenses-processors" ] } ``` From chart 5.5.25, `lenses.sql.namespaces: [lenses-processors]` sets this for you. To let Lenses deploy to any namespace instead, grant the same rules with a `ClusterRole` and `ClusterRoleBinding`, add `list` to the `namespaces` rule, and leave `lenses.kubernetes.namespaces` unset. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.lenses.io/latest/devx/5.5/deployment/configuration/sql-processor-deployment.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.