> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/devx/6.3/deployment/configuration/agent/sql-processor-deployment.md).

# SQL Processor Deployment

This page describes how to configure the agent to deploy and manage SQL Processors for stream processing.

Lenses can be used to define & deploy stream processing applications that read from Kafka and write back to Kafka with SQL. They are based on the Kafka Stream framework. They are known as SQL Processors.

SQL processing of real-time data can run in 2 modes:

* SQL In-Process - the workload runs inside of the Lenses Agent.
* SQL in Kubernetes - the workload runs & scale on your Kubernetes cluster.

Which mode the SQL Processors will run as should be defined within the **lenses.conf** before Lenses is started.

## In-Process Mode <a href="#sql-in-process" id="sql-in-process"></a>

In this mode, SQL processors run as part of the Agent process, sharing resources, memory, and CPU time with the rest of the platform.

{% hint style="warning" %}
This mode of operation is meant to be used for development only.

As such, the agent will not allow the creation of more than 50 SQL Processors in *In Process* mode, as this could impact the platform's stability and performance negatively.

For production, use the `KUBERNETES` mode for maximum flexibility and scalability.
{% endhint %}

Set the execution configuration to `IN_PROC`

```bash
# Set up Lenses SQL processing engine
lenses.sql.execution.mode = "IN_PROC"
```

Set the directory to store the internal state of the SQL Processors:

```bash
lenses.sql.state.dir = "/tmp/sql-kstream-state"
```

### TLS connections to Kafka and Schema Registries <a href="#connections-management" id="connections-management"></a>

SQL processors use the same connection details that Agent uses to speak to Kafka and Schema Registry. The following properties are mounted, if present, on the file system for each processor:

* **Kafka**
  1. SSLTruststore
  2. SSLKeystore
* **Schema Registry**
  1. SSL Keystore
  2. SSL Truststore

The file structure created by applications is the following: **/run/\[lenses\_installation\_id]/applications/**

{% hint style="warning" %}
**Keep in mind**\
Lenses require an installation folder with write permissions. The following are tried:

1. /run
2. /tmp
   {% endhint %}

## Kubernetes Mode <a href="#sql-on-kubernetes" id="sql-on-kubernetes"></a>

Kubernetes can be used to deploy SQL Processors. To configure Kubernetes, set the mode to `KUBERNETES` and configure the location of the kubeconfig file.

When the Agent is deployed inside Kubernetes, the **lenses.kubernetes.config.file** configuration entry should be set to an empty string. The Kubernetes client will auto-configure from the pod it is deployed in.

The SQL Processor docker image is live in Dockerhub.

{% code fullWidth="false" %}

```properties
lenses.sql.execution.mode = KUBERNETES
# kubernetes configuration
lenses.kubernetes.config.file = "/home/lenses/.kube/config"
lenses.kubernetes.service.account = "default"
#lenses.kubernetes.processor.image.name = "lensesioextra/sql-processor" # Only needed if you use a custom image
#lenses.kubernetes.processor.image.tag = "6.3"                           # Only needed if you use a custom image

# Only needed if you want to tune the buffer size for incoming events from Kubernetes
#lenses.deployments.errors.buffer.size = 1000

# Only needed if you want to tune the buffer size for incoming errors from Kubernetes WS communication
#lenses.deployments.events.buffer.size = 10000
```

{% endcode %}

### Allowed images <a href="#allowed-images" id="allowed-images"></a>

The image of a SQL Processor can be changed after it is created. Use `lenses.kubernetes.processor.image.allowed` to control which images it may be changed to. When the setting is not set, any tag or digest of the repository in `lenses.kubernetes.processor.image.name` is allowed.

* An entry without a tag or digest, such as `lensesioextra/sql-processor`, allows any tag or digest of that repository. Names are normalised as Docker does, so `lensesioextra/sql-processor` and `docker.io/lensesioextra/sql-processor` are the same repository.
* An entry with a tag or digest allows exactly that image.
* `"*"` allows any image. It is the only wildcard.
* Entries are matched exactly. An entry is not a prefix: `registry.example.com/mirror` does not cover `registry.example.com/mirror/sql-processor`, and `registry.example.com/mirror/*` is invalid and stops the Agent from starting.

The Agent refuses to start if an entry is not a valid image reference, or if the list does not allow the default image (`lenses.kubernetes.processor.image.name` and `.tag`). For this reason an empty list is not a valid setting. The Agent also refuses to start when `lenses.kubernetes.processor.image.name` and `.tag` together do not form a valid image reference, even if `allowed` is not set. These checks apply when `lenses.sql.execution.mode` is `KUBERNETES`. A change to an image that is not allowed is rejected with a message listing the allowed images.

{% hint style="info" %}
The `lenses.kubernetes.processor.image.allowed` setting is available from Lenses Agent 6.3.1. On earlier versions it is ignored in `lenses.conf`; for the Docker environment variable, see below.
{% endhint %}

This is useful if you mirror the SQL Processor image in a private registry:

{% code title="lenses.conf" fullWidth="false" %}

```properties
lenses.kubernetes.processor.image.allowed = [
  "lensesioextra/sql-processor",
  "registry.example.com/mirror/sql-processor"
]
```

{% endcode %}

When running the Agent in Docker, set the value as a JSON array with each item quoted:

```bash
LENSES_KUBERNETES_PROCESSOR_IMAGE_ALLOWED='["lensesioextra/sql-processor", "registry.example.com/mirror/sql-processor"]'
```

Set this variable only with the Lenses Agent 6.3.1 image or later. Earlier images write the value as a quoted string, and an entry with a tag or digest then makes the configuration invalid, so the Agent won't start.

{% hint style="warning" %}
**Before upgrading to Lenses Agent 6.3.1, allow the images your SQL Processors need.** Without `lenses.kubernetes.processor.image.allowed`, the image of a SQL Processor can only be changed to a tag or digest of the repository in `lenses.kubernetes.processor.image.name`. If you change processors to other images, for example custom images with UDFs or images from another repository or registry, add them to the setting, together with the default repository. In `lenses.conf` you can do this before upgrading; with Docker, set the environment variable in the same change as the 6.3.1 image. For example:

```properties
lenses.kubernetes.processor.image.allowed = [
  "lensesioextra/sql-processor",
  "registry.example.com/team/sql-processor-udfs"
]
```

Existing processors keep their image and can still be started, stopped and scaled; only changing the image is checked.
{% endhint %}

### Security Context <a href="#pod-security-context" id="pod-security-context"></a>

If your cluster enforces Pod Security Standards or an admission policy (e.g. OPA/Gatekeeper) that requires specific pod or container security settings, such as running as non-root or dropping Linux capabilities, you can set a `securityContext` on the SQL Processor pod and on its containers:

{% code title="lenses.conf" fullWidth="false" %}

```properties
lenses.kubernetes.pod.security.context {
  runAsUser = 1001
  runAsGroup = 1001
  runAsNonRoot = true
  fsGroup = 1001
}

lenses.kubernetes.container.security.context {
  capabilities { drop = [ALL] }
  allowPrivilegeEscalation = false
}
```

{% endcode %}

Each block accepts any field the Kubernetes API's `PodSecurityContext` (pod) or `SecurityContext` (container) types support, so it can satisfy whatever a cluster's admission policy requires.

{% hint style="info" %}
**Deprecated: typed processor security context fields**

The older `lenses.kubernetes.processor.security.context.{runAsUser,runAsGroup,runAsNonRoot,fsGroup}` config is deprecated in favour of `lenses.kubernetes.pod.security.context` above, which covers the same fields plus anything else `PodSecurityContext` supports. It keeps working as a fallback when `pod.security.context` isn't set, but new deployments should use the new property.

```properties
lenses.kubernetes.processor.security.context.runAsUser = "1001"
lenses.kubernetes.processor.security.context.runAsGroup = "1001"
lenses.kubernetes.processor.security.context.runAsNonRoot = "true"
lenses.kubernetes.processor.security.context.fsGroup = "1001"
```

`runAsUser` and `runAsGroup` must both be set for this fallback to apply. If either is missing, no pod security context is applied at all.
{% endhint %}

### Custom Serdes <a href="#custom-serde" id="custom-serde"></a>

Custom serdes should be embedded in a new Lenses SQL processor Docker image.

To build a custom Docker image, create the following directory structure:

```bash
mkdir -p processor-docker/serde
```

Copy your serde jar files under processor-docker/serde.

Create `Dockerfile` containing:

```dockerfile
FROM lensesioextra/sql-processor:4.2

ADD serde /opt/serde
ENV LENSES_SQL_RUNNERS_SERDE_CLASSPATH_OPTS=/opt/serde
```

Build the Docker.

```bash
cd processor-docker
docker build -t example/lsql-processor .
```

Once the image is deployed in your registry, please set Lenses to use it (lenses.conf):

```bash
lenses.kubernetes.processor.image.name = "your/image-name"
lenses.kubernetes.processor.image.tag = "your-tag"
```

{% hint style="warning" %}
**Don't use the `LPFP_` prefix.**

Internally, Lenses prefixes all its properties with `LPFP_`.

Avoid passing custom environment variables starting with `LPFP_` as it may cause the processors to fail.
{% endhint %}

## Grant the Agent Kubernetes permissions to deploy processors

The Agent needs Kubernetes API permissions to deploy and manage SQL Processors. There are two ways to grant them:

* **Namespace-scoped `Role`/`RoleBinding` resources — recommended.** The Agent can only deploy into, and read pod/secret information from, the namespace(s) you explicitly list.
* **Cluster-wide RBAC (Helm's `rbacEnable: true` plus `namespaceScope: false`)** — grants access across every namespace in the cluster. Use this only when you cannot enumerate the target namespaces in advance; see [Cluster-wide RBAC](#cluster-wide-rbac-not-recommended) below. Note `namespaceScope` defaults to `true`, so `rbacEnable: true` alone gives you a namespace-scoped Role, not a ClusterRole.

{% hint style="info" %}
These permissions are for the Agent. From Lenses Agent 6.3.1, SQL Processor pods do not mount a Kubernetes API token (`automountServiceAccountToken: false`), since processors do not call the Kubernetes API. They still run as the service account set in `lenses.kubernetes.service.account`, so its image pull secrets still apply. To apply this to processors deployed by an earlier Agent, stop and start them. No configuration change is needed.
{% endhint %}

### Namespace-scoped RBAC (recommended) <a href="#namespace-scoped-rbac-recommended" id="namespace-scoped-rbac-recommended"></a>

Create a **Role** and a **RoleBinding** resource in the namespace you want the processors deployed to:

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: [ROLE_NAME]
  namespace: [PROCESSORS_NAMESPACE]
rules:
# Read-only. `namespaces: get` is required by the Agent's Processor log
# reader; without it, log viewing fails with HTTP 500. A Role bound in this
# namespace granting it permits reading only this namespace object, so it
# does not widen access.
- apiGroups: [""]
  resources:
    - namespaces
    - pods/log
  verbs:
    - get
# Status of SQL Processors and K2K apps. Pods are only created through
# Deployments.
- apiGroups: [""]
  resources:
    - pods
    - events
  verbs:
    - get
    - list
    - watch
# SQL Processor credentials. The Agent never reads a Secret back.
# Agents older than 6.3.1 also need `get` and `update` here.
- apiGroups: [""]
  resources:
    - secrets
  verbs:
    - create
    - delete
# K2K app configuration
- apiGroups: [""]
  resources:
    - configmaps
  verbs:
    - get
    - list
    - watch
    - create
    - update
    - delete
# SQL Processor and K2K workloads
- apiGroups: ["apps"]
  resources:
    - deployments
  verbs:
    - get
    - list
    - watch
    - create
    - update
    - delete
# `deployments/scale` is a separate subresource: granting `deployments` does
# NOT confer it, and the Agent needs it to scale SQL Processors.
- apiGroups: ["apps"]
  resources:
    - deployments/scale
  verbs:
    - get
    - update
```

```yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: [ROLE_BINDING_NAME]
  namespace: [PROCESSOR_NAMESPACE]
subjects:
- kind: ServiceAccount
  namespace: [LENSES_NAMESPACE]
  name: [SERVICE_ACCOUNT_NAME]
roleRef:
  kind: Role
  name: [ROLE_NAME]
  apiGroup: rbac.authorization.k8s.io
```

example for:

* Lenses namespace = `lenses-ns`
* Processor namespace = `lenses-proc-ns`

```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role
  namespace: lenses-proc-ns
rules:
# Read-only. `namespaces: get` is required by the Agent's Processor log
# reader; without it, log viewing fails with HTTP 500. A Role bound in this
# namespace granting it permits reading only this namespace object, so it
# does not widen access.
- apiGroups: [""]
  resources:
    - namespaces
    - pods/log
  verbs:
    - get
# Status of SQL Processors and K2K apps. Pods are only created through
# Deployments.
- apiGroups: [""]
  resources:
    - pods
    - events
  verbs:
    - get
    - list
    - watch
# SQL Processor credentials. The Agent never reads a Secret back.
# Agents older than 6.3.1 also need `get` and `update` here.
- apiGroups: [""]
  resources:
    - secrets
  verbs:
    - create
    - delete
# K2K app configuration
- apiGroups: [""]
  resources:
    - configmaps
  verbs:
    - get
    - list
    - watch
    - create
    - update
    - delete
# SQL Processor and K2K workloads
- apiGroups: ["apps"]
  resources:
    - deployments
  verbs:
    - get
    - list
    - watch
    - create
    - update
    - delete
# `deployments/scale` is a separate subresource: granting `deployments` does
# NOT confer it, and the Agent needs it to scale SQL Processors.
- apiGroups: ["apps"]
  resources:
    - deployments/scale
  verbs:
    - get
    - update
```

```yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: processor-role-binding
  namespace: lenses-proc-ns
subjects:
- kind: ServiceAccount
  namespace: lenses-ns
  name: default
roleRef:
  kind: Role
  name: processor-role
  apiGroup: rbac.authorization.k8s.io
```

{% hint style="info" %}
You can repeat this for as many namespaces you may want Lenses to have access to.
{% endhint %}

{% hint style="warning" %}
**`lenses.kubernetes.namespaces` is required alongside a namespace-scoped `Role`.** Without it, the Agent's SQL Processor watchers issue cluster-wide `list`/`watch` calls, which a namespace-scoped `Role` denies — processor deployment will fail with a permissions error. Granting a `ClusterRole` to work around that error reopens the cross-namespace access the `Role`/`RoleBinding` was meant to prevent. Always set `lenses.kubernetes.namespaces` together with the `Role`/`RoleBinding`, as shown below.
{% endhint %}

Define in the Agent configuration which namespaces it can access. To achieve this amend `values.yaml` to contain the following:

```toml
lensesAgent:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "[PROCESSORS NAMESPACE]"
        ]
      }      
```

example:

```toml
lensesAgent:
  append:
    conf: |
      lenses.kubernetes.namespaces = {
        incluster = [
          "lenses-processors"
        ]
      }      
```

### Cluster-wide RBAC (not recommended) <a href="#cluster-wide-rbac-not-recommended" id="cluster-wide-rbac-not-recommended"></a>

Alternatively, activate RBAC at the cluster level through Helm `values.yaml`. **Both** values are required — `namespaceScope` defaults to `true`, so `rbacEnable: true` on its own produces a namespace-scoped `Role`, not a `ClusterRole`:

```
rbacEnable: true
namespaceScope: false
```

{% hint style="danger" %}
**Cluster-wide RBAC is not recommended for production.** It grants the Agent's service account permissions across every namespace in the cluster: processors can be deployed into any namespace, Secrets can be created in any namespace, and pod information can be read cluster-wide, not just in the namespaces running Lenses workloads. Use it only if you genuinely need processors deployed into namespaces you cannot enumerate in advance — for example, namespaces created dynamically per tenant — and accept that broader exposure.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.lenses.io/latest/devx/6.3/deployment/configuration/agent/sql-processor-deployment.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.