# TimestampConverter An adapted version of the [TimestampConverter](https://github.com/apache/kafka/blob/5c2492bca71200806ccf776ea31639a90290d43e/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TimestampConverter.java#L50) SMT. The SMT adds a few more features to the original: * allows nested fields resolution (e.g. `a.b.c`) * uses \_key or \_value as prefix to understand the field to convert is part of the record Key or Value * allows conversion from one string representation to another (e.g. `yyyy-MM-dd HH:mm:ss` to `yyyy-MM-dd`) * allows conversion using a rolling window boundary (e.g. every 15 minutes, or one hour) ## Transform Type Class ``` io.lenses.connect.smt.header.TimestampConverter ``` ## Configuration | Name | Description | Type | Default | Valid Values | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | ------------ | ------------------------------------------------ | | `header.name` | The name of the header to insert the timestamp into. | String | | | | `field` | The field path containing the timestamp, or empty if the entire value is a timestamp. Prefix the path with the literal string `_key`, `_value` or `_timestamp`, to specify the record Key, Value or Timestamp is used as source. If not specified `_value` is implied. | String | | | | `target.type` | Sets the desired timestamp representation. | String | | string,unix,date,time,timestamp | | `format.from.pattern` | Sets the format of the timestamp when the input is a string. The format requires a Java DateTimeFormatter-compatible pattern. Multiple (fallback) patterns can be added, comma-separated. | String | | | | `format.to.pattern` | Sets the format of the timestamp when the output is a string. The format requires a Java DateTimeFormatter-compatible pattern. | String | | | | `rolling.window.type` | An optional parameter for the rolling time window type. When set it will adjust the output value according to the time window boundary. | String | none | none, hours, minutes, seconds | | `rolling.window.size` | An optional positive integer parameter for the rolling time window size. When `rolling.window.type` is defined this setting is required. The value is bound by the `rolling.window.type` configuration. If type is `minutes` or `seconds` then the value cannot bigger than 60, and if the type is `hours` then the max value is 24. | Int | 15 | | | `unix.precision` | The desired Unix precision for the timestamp. Used to generate the output when type=unix or used to parse the input if the input is a Long. This SMT will cause precision loss during conversions from, and to, values with sub-millisecond components. | String | milliseconds | seconds, milliseconds, microseconds, nanoseconds | | `timezone` | Sets the timezone. It can be any valid java timezone. Overwrite it when `target.type` is set to `date, time, or string`, otherwise it will raise an exception. | String | UTC | | ## Example To convert to and from a string representation of the date and time in the format `yyyy-MM-dd HH:mm:ss.SSS`, use the following configuration: ```properties transforms=TimestampConverter transforms.TimestampConverter.type=io.lenses.connect.smt.header.TimestampConverter transforms.TimestampConverter.header.name=wallclock transforms.TimestampConverter.field=_value.ts transforms.TimestampConverter.target.type=string transforms.TimestampConverter.format.from.pattern=yyyyMMddHHmmssSSS transforms.TimestampConverter.format.to.pattern=yyyy-MM-dd HH:mm:ss.SSS ``` To convert to and from a string representation while applying an hourly rolling window: ```properties transforms=TimestampConverter transforms.TimestampConverter.type=io.lenses.connect.smt.header.TimestampConverter transforms.TimestampConverter.header.name=wallclock transforms.TimestampConverter.field=_value.ts transforms.TimestampConverter.target.type=string transforms.TimestampConverter.format.from.pattern=yyyyMMddHHmmssSSS transforms.TimestampConverter.format.to.pattern=yyyy-MM-dd-HH transforms.TimestampConverter.rolling.window.type=hours transforms.TimestampConverter.rolling.window.size=1 ``` To convert to and from a string representation while applying an hourly rolling window and timezone: ```properties transforms=TimestampConverter transforms.TimestampConverter.type=io.lenses.connect.smt.header.TimestampConverter transforms.TimestampConverter.header.name=wallclock transforms.TimestampConverter.field=_value.ts transforms.TimestampConverter.target.type=string transforms.TimestampConverter.format.from.pattern=yyyyMMddHHmmssSSS transforms.TimestampConverter.format.to.pattern=yyyy-MM-dd-HH transforms.TimestampConverter.rolling.window.type=hours transforms.TimestampConverter.rolling.window.size=1 transforms.TimestampConverter.timezone=Asia/Kolkata ``` To convert to and from a string representation while applying a 15 minutes rolling window: ```properties transforms=TimestampConverter transforms.TimestampConverter.type=io.lenses.connect.smt.header.TimestampConverter transforms.TimestampConverter.header.name=wallclock transforms.TimestampConverter.field=_value.ts transforms.TimestampConverter.target.type=string transforms.TimestampConverter.format.from.pattern=yyyyMMddHHmmssSSS transforms.TimestampConverter.format.to.pattern=yyyy-MM-dd-HH-mm transforms.TimestampConverter.rolling.window.type=minutes transforms.TimestampConverter.rolling.window.size=15 ``` To convert to and from a Unix timestamp, use the following: ```properties transforms=TimestampConverter transforms.TimestampConverter.type=io.lenses.connect.smt.header.TimestampConverter transforms.TimestampConverter.header.name=wallclock transforms.TimestampConverter.field=_key.ts transforms.TimestampConverter.target.type=unix transforms.TimestampConverter.unix.precision=milliseconds ``` Here is an example using the record timestamp field: ```properties transforms=TimestampConverter transforms.TimestampConverter.type=io.lenses.connect.smt.header.TimestampConverter transforms.TimestampConverter.header.name=wallclock transforms.TimestampConverter.field=_timestamp transforms.TimestampConverter.target.type=unix transforms.TimestampConverter.unix.precision=milliseconds ``` ## Configuration for `format.from.pattern` {% hint style="warning" %} Configuring multiple `format.from.pattern` items requires careful thought as to ordering and may indicate that your Kafka topics or data processing techniques are not aligning with best practices. Ideally, each topic should have a single, consistent message format to ensure data integrity and simplify processing. {% endhint %} ### Multiple Patterns Support The `format.from.pattern` field supports multiple DateTimeFormatter patterns in a comma-separated list to handle various timestamp formats. Patterns containing commas should be enclosed in double quotes. For example: ``` format.from.pattern=yyyyMMddHHmmssSSS,"yyyy-MM-dd'T'HH:mm:ss,SSS" ``` ### Best Practices While this flexibility can be useful, it is generally not recommended due to potential complexity and inconsistency. Ideally, a topic should have a single message format to align with Kafka best practices, ensuring consistency and simplifying data processing. ### Configuration Order The order of patterns in `format.from.pattern` matters. Less granular formats should follow more specific ones to avoid data loss. For example, place `yyyy-MM-dd` after `yyyy-MM-dd'T'HH:mm:ss` to ensure detailed timestamp information is preserved. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.lenses.io/latest/connectors/single-message-transforms/timestampconverter.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.