Deploying SQL Processors¶
Lenses allows you to create SQL processors which are Kafka Streams application written in Lenses SQL.
The user experience is similar for each execution mode, however
KUBERNETES offer the
ability to scale the number of processors up or down and have additional properties such as the cluster and
namespace to deploy to.
Create a Lenses SQL Processor¶
Navigate to the
SQL Processors view from the side menu. This view will list and provide details for the currently
deployed processors. The
execution mode is displayed on the top right alongside the button for creating new processors.
New Processor option will present you with a modal form allowing you to configure the processor.
Each processor requires:
- Name - This must be unique.
- Runners - The parallelization, how many instances of the Lenses SQL application you want running.
- Lenses SQL statement - The SQL code for the stream processing.
- Cluster - The cluster to deploy to, either the Connect cluster or Kubernetes cluster. Only applicable
- Namespace - Kubernetes namespace to deploy to. Only applicable to the
- Pipeline - Optional label for Kubernetes which is attached to the pods. Kubernetes will inject this into metrics collected by Prometheus.
The image below shows an example for Kubernetes.
KUBERNETES execution mode, it may take a few seconds for Kubernetes to pull and deploy the Lenses SQL docker image
and the runners to report in.
Status and Configuration¶
Processors status and metrics are tracked in real time. Selecting a processor will provide its topology graph generated by the SQL engine. Interacting with the input/output topic(s) nodes on the topology graph drives will end up displaying the data flowing through them.
Check the metrics in order to get an understanding of how the Lenses SQL processor is performing. This includes the lag for the source topics.
Lenses allows you to stop, start, scale and delete processors.
Select the desired processor to go to the detailed view and then use the
Actions button for the list of available operations.
When using the
KUBERNETES execution mode, it might take a few seconds for Kubernetes to pull and deploy the Lenses SQL docker image
and the runners to report in, or for Kubernetes to terminate and clean up the pods when scaling down.
Any change in a processors configuration is audited and a Last Action message is also displayed.
You might want to control who makes changes to your processor. By default, processor creation, deletion and editing is only allowed for users with WRITE access, such as admin and write.