Creating and deleting SQL Processors

Navigate to Apps to view, manage and create new processors.

SQL processors list

Create new Processor 

To create a new processor navigate to Apps, and click the New App => SQL Processor. You will be prompt with the form to fill in.

NameA friendly name (any string)
RunnersNumber of runners based on your deployment targets
SQLYou need to follow the Lenses SQL Streaming syntax
ProcessorIDAbout processorID
SQL processor create inproc

When creating a new processor it will not start automatically. You will have to explicitly start the processor in order to start consuming data. When you hit the start button, and depending on your deployment target will deploy the number of runners


ProcessorID is the public unique identifier for an SQL processor. You can customize to any arbitrary string following the bellow rules:

  • Must be unique across all processors
  • Match the following regex: ^[a-zA-Z0-9\-\._]+
    • Only letters, capital letters, numbers and -, _ & - are allowed.
    • It has to start with a letter or a number.
    • It cannot be empty, if empty will be filled automatically
Screenshot Processor ID

ProcessorID is used as the Kafka consumer group identifier. That means that you can instruct an SQL processor to build its consumer group and coordinate record ingestion from Kafka between all Processor replicas. Consequently, if the ProcessorId of a given SQL processor is changed, that processor will restart consuming messages from the beginning of the existing records in the topic.

You can set it when creating a new processor:

SQL processor ID


The ApplicationID is the Lenses unique identifier, is automatically created by Lenses, and cannot be customized. This is unique among all SQL Processors.

Lenses uses the ApplicationID to manage applications. This means that, when Starting, Stopping or Scaling an application, Lenses will use this attribute to pick the right instance.

While it’s an internal id, you can still refer to it via the UI:

SQL processor application id

Monitor Processor 


You can monitor your processor metrics to ensure that data is processed as expected and all runners are working. You can see specific metrics for data consumption and scale up or down according to your needs or resources capacity.

sql processor detail monitor

Processor Topology 

After creating a processor, the topology view of your SQL will be generated to visualize how the data is processed and what are the steps to produce the output. The nodes are interactive, so you can click and get some more insights on each stage.

sql processor detail

Underlying consumer

As the processor consumes data from your topics you can navigate to the underlying Kafka Consumer to set alerts for lag or view more granular information about the consumption. You can use the processorID to detect the processor under the consumers page.

You can also manage the offsets ( replay, skip etc ) from the consumer group, but the processor should not be running. For consumers you will need additional permissions.

SQL processor consumer change offset

Delete Processors 

To delete a running processor you will need to first Stop it from running. Then the delete option will be available and you will be able to remove the processor.

Sql Processor delete

When deleting a processor deployed in an external target ie. Kubernetes, with one or multiple runners, all runners (ie. relevant pods) will be removed.

If the processor has created topics or schemas as part of its execution, they are not going to be deleted. You can use the processorID to search for relevant topics and schemas as they will be prefixed.

Scale Runners 

A SQL Processor is a continuously running process and can be deployed as an application based on the deployment target configured to Lenses. If you haven’t configured a deployment target you can still run SQL Processors “in process” but you will be limited in scaling the app and the number of processors you can create.

Scaling means that the application will run with multiple runners to share the load, be resilient and fault tolerant. The maximum number of runners would be the maximum number of partitions your processor consumes.

To scale the runners, you will need to have Lenses configured with a deployment target ie. Kubernetes. Then from the UI you can select the number fo runners to scale up or down:

SQL processors scale button
Sql processors update runners

New runners will start, or existing will stop according to the desired state.

SQl processor runners

You can click on each runner to get specific performance metrics about consumption, production, errors etc:

SQL processor specific runner

Manage older Processors (<v.4.0) 

Since version 4.0 a new generation of the SQL streaming engine is available in order to create SQL Processors. The new version brought a number of new capabilities, it’s more robust and flexible. The new versions of Lenses fully support the new engine but it is not compatible with older versions of SQL Processors. That means that if you have SQL processors running on version < 4.0.x you can still have some basic capabilities from Lenses interface, but it’s recommended that you migrate to the new SQL streaming.

Deprecated Processors 

All the processors that have been created with Lenses 3.2 and older will be marked as “Deprecated” so you can identify and plan to migrate.

deprecated processors

If you have processors running in on of the external targets they will automatically be imported:

  • In KUBERNETES, since version 4.0

The processors can still be manged via Lenses interface (Start, Stop, Scale, Delete), but the topology graph will not be plotted with the detailed nodes:

Deprecated processors details

The “Processor Graph” of the Deprecated Processor will now be represented by only one(1) node which is called “PROCESSOR”

 Deprecated processors details topology