Keywords

Lenses SQL uses MySQL syntax for dual purposes:

  • read data from a topic [and partition/-s] while applying the required filters and functions on fields.
  • build and run an Apache Kafka Streams application by using a simple SQL like syntax.

To support the particular constructs of a Kafka record the engine is using the following set of keywords:

Name Description Optional Type
_key

Is used as a prefix and instructs LSQL to pick up the field from the Kafka message key part.
 YES  
_ktype

Specifies what decoder to use for the key component when reading from the given topic.

 NO STRING
_vtype

Specifies what decoder to use for the value component when reading from the given topic.

 NO STRING
_partition

Specifies which partitions to read from. It can be _partition=2 or _partition in (1,2,3).
If used in a SELECT field list it returns the Kafka message partition number.
 YES INT
_offset

Specifies which offset to read from. It can be bounded by a specific range or a specific value.
If _offset=123 one message is read. The user can restrict further by specifying lower/upper
bound.``_offset >= 123 AND _offset < 999``. If used in a select gives back the Kafka offset.
 YES INT
_ts

Restricts the messages read based on their timestamp. Similar to offset it can be bounded
by a specific range or a specific value _ts=1543366 or a range _ts > 123 AND _ts < 125.
Please consult Apache Kafka documentation for timestamp information. If used in a SELECT
statement it returns the Kafka message timestamp.
 YES LONG
_value Is used as a prefix and instructs LSQL to return the value of a Kafka message value part. YES  

To address the headers set on a record the engine provides specific functions. Jump to the functions documentation and look for headerAs* method/s.

The SQL is using a grammar which has the following keywords predefined:

  • timestamp
  • partition
  • topic
  • date

These keywords can still be used as fields, but they must be escaped. It is recommended, as with most SQL dialects, to escape field and topic names. Below is an example of such query:

SELECT
  `partition`
  , `timestamp`
FROM `target_topic`

Important

Always escape the topic name via `. If you don’t and the topic contains non-alphanumeric characters the parsing will fail.