Overview
This page describes an overview of the Lenses SMTs for Kafka Connect.
Last updated
Was this helpful?
This page describes an overview of the Lenses SMTs for Kafka Connect.
Last updated
Was this helpful?
Lenses provides several SMTs designed for use with Stream Reactor connectors, you can also use them with other connectors or your own.
These SMTs are designed to be used with the framework. The SMTs create record headers. The advantage of using headers is that they reduce the memory and CPU cycles required to change the payload. See for example the Kafka Connect . Furthermore, they support sink partitioners, for scenarios like:
Partitioning by the system clock (e.g. using the system clock as a partition key with a yyyy-MM-dd-HH format)
Partitioning by a rolling window (e.g. every 15 minutes, or one hour)
Partitioning by a custom timestamp (e.g. a timestamp field in the payload, record Key or Value)
Partitioning by a custom timestamp with a rolling window (e.g. a timestamp field in the payload, every 15 minutes, or one hour)
Add the plugin to the worker classloader isolation via the plugin.path option:
For MSK connect you need to bundle your SMT with the connector you need to use and deploy as one plugin.
This zip (containing both jars at the same level) must be uploaded as a plugin in MSK connect.
To do so, download your connector to get the jar (unzip if needed), download (unzip it). Put both jar in the same folder, and zip them altogether.
