4.3

You are viewing documentation for an older version of Lenses.io View latest documentation here

Meta Fields

When running queries against Kafka, Snapshot Engine enables you to access the record metadata through the special _meta facet.

These are the available meta fields:

FieldDescription
_meta.offsetThe offset of the record in its Kafka topic partition
_meta.partitionThe Kafka topic partition of the record
_meta.timestampThe Kafka record timestamp
_meta.__keysizeThe length in bytes of the raw key stored in Kafka
_meta.__valuesizeThe length in bytes of the raw value stored in Kafka

Examples 

Select all the meta fields 

The following query will select all the meta fields listed above:

SELECT _meta.* FROM topic

Filter on record timestamp 

SELECT ...
    FROM topic
WHERE _meta.timestamp > YESTERDAY()

Filter on table partition 

To read records from a specific partition, the following query can be used:

SELECT ...
    FROM topic
WHERE _meta.partition = 1
    OR _meta.partition = 8

Search for a record on a specific offset 

Here is the query to use when the record offset and partition are known:

SELECT ...
    FROM topic
WHERE _meta.partition = 2
    AND _meta.offset = 8
LIMIT 1

Get the latest N records per partition 

This query will get the latest 100 records per partition (assuming the topic is not compacted):

SELECT ...
    FROM topic
WHERE _meta.offset >= LAST_OFFSET() - 100

This instead will get the latest 100 records for a given partition (again assuming the topic is not compacted):

SELECT ...
    FROM topic
WHERE _meta.offset >= LAST_OFFSET() - 100 
    AND _meta.partition = 2