Deploying SQL Processors¶
Lenses allows you to create LSQL processors which are Kafka Streams application written in LSQL.
The user experience is similar for each execution mode, however
KUBERNETES offer the
ability to scale the number of processors up or down and have additional properties such as the cluster and
namespace to deploy to.
Create a LSQL Processor¶
Navigate to the
SQL Processors view from the side menu. This view will list and provide details for the currently
deployed processors. The
execution mode is displayed on the top right alongside the button for creating new processors.
New Processor option, this will present you with a modal form allowing you to configure the processor.
Each processor requires:
- Name - This must be unique
- Runners - The parallelization, how many instances of the LSQL application you want running
- Lenses SQL statement - The LSQL code for the stream processing
- Cluster - The cluster to deploy to, either the Connect cluster or Kubernetes cluster. Only applicable
- Namespace - Kubernetes namespace to deploy to. Only applicable to the
- Pipeline - Optional label for Kubernetes which is attached to the pods. Kubernetes will inject this into metrics collected by Prometheus.
The image below shows an example for Kubernetes.
In execution mode
KUBERNETES it may take a few seconds for Kubernetes to pull and deploy the LSQL docker image
and the runners to report in.
Status and Configuration¶
Processors status and metrics are tracked in real time. Selecting a processor will provide its topology graph generated by the LSQL engine. Interacting with the input/output topic(-s) nodes on the topology graph drives will end up displaying the data flowing through them.
Check the metrics to get an understanding of how the LSQL processor is performing. This includes the lag for the source topics.
Lenses allows you to stop, start, scale and delete processors.
Select the desired processor to go to the detailed view and then use the
Actions button for the list of available operations.
KUBERNETES execution mode, it may take a few seconds for Kubernetes to pull and deploy the LSQL docker image
and the runners to report in, or for Kubernetes to terminate and clean up the pods when scaling down.
Any change in a processors configuration is audited and a Last Action message is also displayed.
You might want to control who makes changes to your processor. By default, processor creation, deletion and editing is only allowed for users with WRITE access, such as admin and write.