Process data with SQL Processors

Understand how to do streaming processing with Lenses SQL Processors on Apache Kafka

SQL Processors are long-lived applications that continuously process Kafka data. In Kubernetes mode, which is the recommended setup, Lenses deploys them as separate Kubernetes workloads.

SQL Processors run continuous SQL queries on Kafka topics. They are built on Kafka Streams and managed through Lenses, so you can create, deploy, monitor, and scale streaming applications from one place.

Use SQL Processors when data should be handled as it arrives. For example, you can filter records, join streams, aggregate metrics, and transform topics into new outputs.

SQL Processors offer:

• a no-code application that runs a Lenses SQL query on current and future data

• topology visualization

• integrated lifecycle management in Lenses

• monitoring and alerting support

• horizontal scaling through Kubernetes deployments

Create a SQL Processor

Open Apps from the left navigation. Then open SQL Processors and select Create SQL Processor. You can also create one from the SQL Processors node in the navigation tree.

1

Add metadata

Enter a processor name, then optionally add a description and tags.

2

Select the environment and namespace

Select the environment where the processor will run. In Kubernetes mode, also select the namespace.

3

SQL

Write the continuous SQL query that reads from source topics and writes to output topics.

You can also configure:

1. Runners — the number of processor instances to deploy. Use auto max to size the runner count from the source topics.

2. Processor ID — the public identifier for the processor. This value also controls the Kafka consumer group used by the processor. See Processor ID and Application ID.

3. Kubernetes settings — override the default SQL Processor image, set CPU and memory requests or limits, and add extra pod labels.

Write SQL with auto-complete

The editor provides auto-complete, validation, and inline errors while you write the query.

Use Ctrl + Space or Cmd + Space for auto-complete. Use it again to inspect more detail for topics and fields.

For the query model and SQL semantics, see Concepts. For runtime options, see Settings.

Processor ID and Application ID

The processor has two important identifiers.

Processor ID is the user-facing unique identifier for a SQL Processor. You control this value.

Restrictions on custom Processor IDs:

• it must be unique across all SQL Processors

• it must match ^[a-zA-Z0-9\-\._]+

  • only letters, numbers, ., _, and - are allowed

  • it must start with a letter or number

  • it cannot be empty

The Processor ID is also used as the Kafka consumer group identifier for the processor. That means it controls how replicas coordinate ingestion from Kafka. If you change the Processor ID, the processor uses a new consumer group and will typically read from the beginning unless offsets already exist for that new ID.

Application ID is the internal identifier generated by Lenses. You cannot change it.

It is unique across Lenses applications and is used internally when Lenses starts, stops, scales, and tracks the processor.

View and manage SQL Processors

Open Apps from the left navigation, then open SQL Processors. You can also expand a target environment and open its SQL Processor nodes there.

Select a processor from the list or from the navigation tree to open its details.

Each processor has these sections. You can open them from breadcrumbs, from the navigation tree, or from context menus.

1. Profile — view metadata, summary metrics, and the topics involved. Topic entries include context actions for deeper inspection.

2. SQL — view the SQL currently running. You cannot edit a processor in place because changes can affect internal state and backing topics.

3. Health — inspect metrics, runner-level status, stack traces, and logs.

4. Alerts — view alert events and manage alert rules. See Alerting & Monitoring.

5. Topology — inspect the processor topology and follow linked topics through context actions.

6. Activity — inspect the audit trail for the processor. See Governance.

Start, stop, scale, or delete a processor

Use the processor tab context menu or the processor node in the navigation tree to:

1. start a stopped processor

2. stop a running processor

3. scale the runner count up or down

4. delete the processor

Stopping or deleting a processor interrupts active processing. Scaling changes the number of active replicas and can trigger consumer group rebalancing.