Tunning

The tabled based engine has a few parts which can be tuned. From controlling the resources involved when running a query to tweaking for performance the underlying Kafka records consumption, it can all be understood in this chapter. To instruct the engine what to do, a user would have to set a few parameters via the SET statement.

Control execution

Querying a table could take a long time if it contains a lot of records. The underlying Kafka topic has to be read and the filter conditions applied and the projections made. Even more, the SELECT statement could end up bringing a large amount to the client. To be able to constrain the resources involved the SQL engine allows for context customization, which would end up driving the execution thus giving the control to the user. Here is the list of context parameters to overwrite:

Name Description Usage
max.size
The maximum amount of data to return.
It can be expressed as bytes (1024b), as kilo-bytes (1024k)
, as mega-bytes (10m) or as giga-bytes (5g).
Default is 20MB
SET max.bytes = '20m';
will set a max of 20MB to be returned
max.query.time
The maximum amount of time the query
is allowed to run. It can be specified as millisecon (2000ms)
,as hours (2h), minutes (10m) or seconds (60s).
Default is 1 hour
SET max.time = '60000ms';
sets a one minute query limit
max.idle.time
The amount of time to wait
when no more records are read from the source
before the query is completed.
Default is 5 seconds.
SET max.idle.time = '5s';;
sets a maximum of 5 calls returning 0.
LIMIT N
The maximum of records to return.
Default is 10000
SELECT *
FROM payments
LIMIT 100;

Query tuning

Lenses SQL uses Kafka Consumer to read the data. This means, an advanced user with knowledge of Kafka, could tweak the consumer properties to achieve better throughput. This would occur on very rare occasions. The query context can receive Kafka consumer settings. For example, to set the consumer max.poll.records can be set as:

SET max.poll.records= 100000;
SELECT *
FROM payments
LIMIT 1000000