Managing queries

Understand to manage queries for Lenses SQL Snapshot concepts engine for Apache Kafka

Termination Control

SELECT * FROM topicA WHERE _key.deviceId=123 LIMIT 10

Adding a LIMIT 10 in the SQL query will result in the SQL terminating early, as soon as 10 x messages have been discovered. It’s not a perfect solution as we might never find 10 x messages, and thus perform a full scan.

You can also set a maximum query or idle time:

SET max.query.time = 30s;

or max idle time, the idea is that there is no reason to keep polling if we have exhausted the entire topic:

SET max.idle.time = 5s;

You can also set the maximum Kafka data scanned per query. max.size is expressed as bytes. It is not a memory allocation, and it does not set the number of returned records. LIMIT sets the returned-record limit, but a query can reach max.size before it finds enough matching records.

SET max.size = '200m';

If SQL Studio shows Query is finished, max size reached, the query reached its configured Kafka byte-scan cap. It did not reach a row limit or memory limit. Set max.size to '200m', then rerun the query to allow up to 200 MB of Kafka data to be scanned.

Use the smallest cap that meets your need. Raising the cap can increase Kafka and broker processing, network bandwidth between Kafka and Lenses, and response time. Narrow the query with topic, partition, offset, or timestamp filters where possible.

Recent queries

Recent queries are displayed, but only for the current session, they are not currently retained.

Click on the play button to run a previous query. If a query is already running, you will be asked if you want to stop it first.

View All queries

SHOW ALL QUERIES

View Running queries

You can see all running queries by Lenses users using SQL:

SHOW QUERIES

Kill Running queries

You can force stop a query by another user using SQL:

KILL QUERY <id>

Concurrent queries limit

Starting with 6.1.3 version, there's a global limit on concurrent queries, as well as a limit per user. They can be set via the agent configuration:

lenses.sql.settings.max.concurrent.queries=200
lenses.sql.settings.max.concurrent.queries.per.user=2