SQL Processors

This page describes creating and managing SQL Processors in Lenses.

SQL Processors continuously process data in Kafka topics. Under the hood, the Kafka Streams APIs are combined with the internal Lenses application deployment framework. A user gets a seamless experience for creating scalable processing components to filter, aggregate, join, and transform Kafka topics, which can scale natively to Kubernetes.

SQL Processors are continuous queries, they process data as it arrives and react to events, e.g. payment made, $5 to Coffeshop. The data is active, the query is passive.

SQL Processors offer:

• A no-code, stand-alone application executing a given Lenses SQL query on current and future data

• Query graph visualisation

• Fully integrated experience within Lenses

• ACLs and Monitoring functionality out-of-the-box

• Ability to scale up and down workflows via Kubernetes

SQL Processors are long-lived applications that continuously process data. In use with Kubernetes mode (recommended), they are deployed via Lenses, as Kubernetes deployments separate from the Lenses instance.

Creating a SQL Processor

To create SQL Processors go to Environments->[Your Environment]->Workspace->Apps->New App->SQL Processor.

1. Enter a name

2. Enter your SQL statement, the editor will help you with IntelliSense

3. Optionally specify a description, tag and ProcessorID (consumer group ID)

4. Select the deployment target, Kubernetes this will be a Kubernetes cluster and namespace.

If the configuration is valid the SQL Processor will be created. You can then click Start to deploy. Processors are not started automatically, click Start in the Actions menu to start the Processor.

Starting a SQL Processor

To start a Processor, select Start from the Action menu.

Viewing SQL Processors

Select the SQL processor. Lenses will show an overview of the health of the processor.

Selecting a Runner will display further information about each individual runner.

The metadata in the Summary tab also contains the consumer group ID (Processor ID).

The SQL tab shows the SQL Streaming statement the processor is running. A visual representation is shown in the Data Flow tab.

The Configuration tab shows the low-level settings.

Scaling a SQL Processor

To scale a processor, Select Scale from the Actions menu, and enter the desired amount of runners. In Kubernetes mode, the runners are pods.

Stopping a SQL Processor

Select Stop in the Actions Menu.

Snippets

Lenses provides helpful snippets for common scenarios. Select the snippet from the Help section.

Processor ID vs Application ID

ProcessorID is the public unique identifier for an SQL processor. It is customizable, meaning that you, as a user, have control over it and can set this identifier to any arbitrary string.

Restrictions on custom ProcessorIDs:

• They have to be unique across all processors

• Match the following regex: ^[a-zA-Z0-9\-\._]+:

  • Only letters, capital letters, numbers and -, _ & - are allowed.

  • It has to start with a letter or a number.

  • It cannot be empty.

One important aspect of the ProcessorID is that it is used as the Kafka consumer group identifier. That means that, in practice, this is the value that allows an SQL processor to build its consumer group and coordinate record ingestion from Kafka between all Processor replicas. Consequently, if the ProcessorID of a given SQL processor is changed, that processor will restart consuming messages from the beginning of the existing records in the topic.

The ApplicationID is the Lenses unique identifier, is automatically created by Lenses, and cannot be customized.

This is unique among all applications types; it does not matter if it’s an SQL processor or a different (new) sort of future application.

Lenses uses the ApplicationID to manage applications. This means that, when Starting, Stopping or Scaling an application, Lenses will use this attribute to pick the right instance.