Keywords¶
Lenses SQL uses MySQL syntax for dual purposes:
- read data from a topic [and partition/-s] while applying the required filters and functions on fields.
- build and run an Apache Kafka Streams application by using a simple SQL like syntax.
To support the particular constructs of a Kafka record the engine is using the following set of keywords:
Name | Description | Optional | Type |
---|---|---|---|
_key | Is used as a prefix and instructs LSQL to pick up the field from the Kafka message key part.
|
YES | |
_ktype | Specifies what decoder to use for the
key component when reading from the given topic. |
NO | STRING |
_vtype | Specifies what decoder to use for the
value component when reading from the given topic. |
NO | STRING |
_partition | Specifies which partitions to read from. It can be
_partition=2 or _partition in (1,2,3) .If used in a SELECT field list it returns the Kafka message partition number.
|
YES | INT |
_offset | Specifies which offset to read from. It can be bounded by a specific range or a specific value.
If
_offset=123 one message is read. The user can restrict further by specifying lower/upperbound.``_offset >= 123 AND _offset < 999``. If used in a select gives back the Kafka offset.
|
YES | INT |
_ts | Restricts the messages read based on their timestamp. Similar to
offset it can be boundedby a specific range or a specific value
_ts=1543366 or a range _ts > 123 AND _ts < 125 .Please consult Apache Kafka documentation for timestamp information. If used in a SELECT
statement it returns the Kafka message timestamp.
|
YES | LONG |
_value | Is used as a prefix and instructs LSQL to return the value of a Kafka message value part. | YES |
To address the headers set on a record the engine provides specific functions. Jump to the functions documentation and look for headerAs* method/s.
The SQL is using a grammar which has the following keywords predefined:
- timestamp
- partition
- topic
- date
These keywords can still be used as fields, but they must be escaped. It is recommended, as with most SQL dialects, to escape field and topic names. Below is an example of such query:
SELECT
`partition`
, `timestamp`
FROM `target_topic`
Important
Always escape the topic name via `
. If you don’t and the topic contains non-alphanumeric characters the parsing will fail.