Single Message Transforms

This page contains the release notes for Single Message Transforms.

1.3.2

Adds support for multiple "from" patterns.

This converts the format.from.pattern field in the following SMTs:

• InsertFieldTimestampHeaders

• InsertRollingFieldTimestampHeaders

• TimestampConverter

into a List (comma separated) so that these SMTs can support multiple (fallback) DateTimeFormatter patterns should multiple timestamps be in use.

Configuration Compatibility

When updating your configuration, if format.from.pattern contains commas, enclose the pattern in double quotes.

Configurations should be backwards-compatible with previous versions of the SMT, the exception is if commas are used in the format.from.pattern string.

To update the configuration of format.from.pattern ensure you enclose any pattern which contains commas in double quotes.

Old Configuration:

format.from.pattern=yyyy-MM-dd'T'HH:mm:ss,SSS

New Configuration

format.from.pattern="yyyy-MM-dd'T'HH:mm:ss,SSS"

Multiple format.from.pattern can now be configured, each pattern containing a comma can be enclosed in double quotes:

format.from.pattern=yyyyMMddHHmmssSSS,"yyyy-MM-dd'T'HH:mm:ss,SSS"

Configuration Order

When configuring format.from.pattern, the order matters; less granular formats should follow more specific ones to avoid data loss. For example, place yyyy-MM-dd after yyyy-MM-dd'T'HH:mm:ss to ensure detailed timestamp information isn't truncated.

1.3.1

Increase error information for debugging.

1.3.0

Adds support for adding metadata to kafka connect headers (for a source connector).

1.2.1

Workaround for Connect runtime failing with unexplained exception where it looks like the static fields of parent class is not resolved prop.

1.2.0

Adds support for inserting time based headers using a Kafka message payload field.

1.1.2

Fix public visibility of rolling timestamp headers.

1.1.1

Don't make CTOR protected.

1.1.0

Introducing four new Single Message Transforms (SMTs) aimed at simplifying and streamlining the management of system or record timestamps, along with support for rolling windows. These SMTs are designed to significantly reduce the complexity associated with partitioning data in S3/Azure/GCS Sink based on time, offering a more efficient and intuitive approach to data organization. By leveraging these SMTs, users can seamlessly handle timestamp-based partitioning tasks, including optional rolling window functionality, paving the way for smoother data management workflows.