View the latest documentation 5.5
Lenses SQL processors can be deployed in two different ways:
You can only have one target. To see how to configuration the deployment mode please view the installation section.
This is the default execution where processors run locally in the same process of Lenses. It’s recommended for development or testing as it’s not scalable and tight to the Lenses deployment (ie. when Lenses restarts so is the processor etc. ). There is a maximum limit of 50 IN_PROC processors.
This will deploy and run the processor to a Kubernetes cluster namespace, and can scale with multiple runners. You will only need to create the SQL statements and select your cluster and then Lenses will handle the deployment of the runners and monitor the performance. This is the most recommended way to run processors with Lenses. You will be able to select any configured cluster or namespaces, currently not bound to your permissions group.
With Lenses 5.0 advanced settings can be configured for the Kubernetes pod:
At the current versions of Lenses you cannot change deployment target from the UI, but you can use Lenses CLI to do some manual steps.
Let’s say for example you want to change a deployment target from “in process” to “kubernetes”, then additional information need to be provided ie. the cluster, the namespace etc.
As a pre-requisite you will have to change Lenses configuration to reflect the desired target.
Step 1: List existing processors
lenses-cli processors list
Step 2: Stop all running processors
lenses-cli processor stop --id <application_id>
Step 3: Export processors to generate the YAML files
lenses-cli export processors
You can also export specific processors:
lenses-cli export processors --id <application_id>
In any case, an apps/sql folder will be created, containing a processor descriptor in YAML format
The YAML file looks like this
name: processor-1 sql: |- SET defaults.topic.autocreate=true; INSERT INTO sink-topic SELECT STREAM * FROM source-topic; runnerCount: 1 cluster: IN-PROC namespace: Lenses pipeline: processor-1 appId: lsql-processor-1-1551637290
Step 4: Update the YAML file with Kubernetes details
The updated YAML looks like this:
name: processor-1 sql: |- SET defaults.topic.autocreate=true; INSERT INTO sink-topic SELECT STREAM * FROM source-topic; runnerCount: 1 cluster: my-custom-cluster namespace: sql-lenses pipeline: processor-1 appId: lsql-processor-1-1551637290
Step 5: Re-create the processors
lenses-cli processor create ./apps/lenses-sql/*
Step 6: Listing processors again will have the new deployment target
Step 7: The processors are not started so you can now start them
lenses-cli processor start --id <application_id>
On this page