Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
This page describes an overview of the Lenses SMTs for Kafka Connect.
Lenses provides several SMTs designed for use with Stream Reactor connectors, you can also use them with other connectors or your own.
These SMTs are designed to be used with the Kafka Connect framework. The SMTs create record headers. The advantage of using headers is that they reduce the memory and CPU cycles required to change the payload. See for example the Kafka Connect TimestampConverter. Furthermore, they support Stream-Reactor S3 sink partitioners, for scenarios like:
Partitioning by the system clock (e.g. using the system clock as a partition key with a yyyy-MM-dd-HH format)
Partitioning by a rolling window (e.g. every 15 minutes, or one hour)
Partitioning by a custom timestamp (e.g. a timestamp field in the payload, record Key or Value)
Partitioning by a custom timestamp with a rolling window (e.g. a timestamp field in the payload, every 15 minutes, or one hour)
Add the plugin to the worker classloader isolation via the plugin.path option:
For MSK connect you need to bundle your SMT with the connector you need to use and deploy as one plugin.
To do so, download your connector to get the jar (unzip if needed), download the SMT (unzip it). Put both jar in the same folder, and zip them altogether.
This zip (containing both jars at the same level) must be uploaded as a plugin in MSK connect.
This page describes how to used SMTs your Kafka Connect Clusters.
Inserts the datetime as a message header from a value field.
This Kafka Connect Single Message Transform (SMT) facilitates the insertion of date and time components (year, month, day, hour, minute, second) as headers into Kafka messages using a timestamp field within the message payload. The timestamp field can be in various valid formats, including long integers, strings, or date objects. The timestamp field can originate from either the record Key or the record Value. When extracting from the record Key, prefix the field with _key.; otherwise, extract from the record Value by default or explicitly using the field without prefixing. For string-formatted fields, specify a format.from.pattern parameter to define the parsing pattern. Long integer fields are assumed to be Unix timestamps; the desired Unix precision can be specified using the unix.precision parameter.
The headers inserted are of type STRING. By using this SMT, you can partition the data by yyyy-MM-dd/HH or yyyy/MM/dd/HH, for example, and only use one SMT.
The list of headers inserted are:
date
year
month
day
hour
minute
second
All headers can be prefixed with a custom prefix. For example, if the prefix is wallclock_, then the headers will be:
wallclock_date
wallclock_year
wallclock_month
wallclock_day
wallclock_hour
wallclock_minute
wallclock_second
When used with the Lenses connectors for S3, GCS or Azure data lake, the headers can be used to partition the data. Considering the headers have been prefixed by _, here are a few KCQL examples:
To use the record Value field named created_at as the unix timestamp, use the following:
To use the record Key field named created_at as the unix timestamp, use the following:
To prefix the headers with wallclock_, use the following:
To change the date format, use the following:
To use the timezone Asia/Kolkoata, use the following:
To facilitate S3, GCS, or Azure Data Lake partitioning using a Hive-like partition name format, such as date=yyyy-MM-dd / hour=HH, employ the following SMT configuration for a partition strategy.
and in the KCQL setting utilise the headers as partitioning keys:
format.from.patternConfiguring multiple format.from.pattern items requires careful thought as to ordering and may indicate that your Kafka topics or data processing techniques are not aligning with best practices. Ideally, each topic should have a single, consistent message format to ensure data integrity and simplify processing.
The format.from.pattern field supports multiple DateTimeFormatter patterns in a comma-separated list to handle various timestamp formats. Patterns containing commas should be enclosed in double quotes. For example:
While this flexibility can be useful, it is generally not recommended due to potential complexity and inconsistency. Ideally, a topic should have a single message format to align with Kafka best practices, ensuring consistency and simplifying data processing.
The order of patterns in format.from.pattern matters. Less granular formats should follow more specific ones to avoid data loss. For example, place yyyy-MM-dd after yyyy-MM-dd'T'HH:mm:ss to ensure detailed timestamp information is preserved.
SMT that inserts date, year, month, day, hour, minute and second headers using the record timestamp. If the record timestamp is null, the SMT uses the current system time.
The headers inserted are of type STRING. By using this SMT, you can partition the data by yyyy-MM-dd/HH or yyyy/MM/dd/HH, for example, and only use one SMT.
The list of headers inserted are:
date
year
month
day
hour
minute
second
All headers can be prefixed with a custom prefix. For example, if the prefix is wallclock_, then the headers will be:
wallclock_date
wallclock_year
wallclock_month
wallclock_day
wallclock_hour
wallclock_minute
wallclock_second
When used with the Lenses connectors for S3, GCS or Azure data lake, the headers can be used to partition the data. Considering the headers have been prefixed by _, here are a few KCQL examples:
To store the epoch value, use the following configuration:
To prefix the headers with wallclock_, use the following:
To change the date format, use the following:
To use the timezone Asia/Kolkoata, use the following:
To facilitate S3, GCS, or Azure Data Lake partitioning using a Hive-like partition name format, such as date=yyyy-MM-dd / hour=HH, employ the following SMT configuration for a partition strategy.
and in the KCQL setting utilise the headers as partitioning keys:
Kafka SMT that inserts date, year, month, day, hour, minute and second headers using the system timestamp and a rolling time window configuration.
The headers inserted are of type STRING. By using this SMT, you can partition the data by yyyy-MM-dd/HH or yyyy/MM/dd/HH, for example, and only use one SMT.
The list of headers inserted are:
date
year
month
day
hour
minute
second
All headers can be prefixed with a custom prefix. For example, if the prefix is wallclock_, then the headers will be:
wallclock_date
wallclock_year
wallclock_month
wallclock_day
wallclock_hour
wallclock_minute
wallclock_second
When used with the Lenses connectors for S3, GCS or Azure data lake, the headers can be used to partition the data. Considering the headers have been prefixed by _, here are a few KCQL examples:
To store the epoch value, use the following configuration:
To prefix the headers with wallclock_, use the following:
To change the date format, use the following:
To use the timezone Asia/Kolkoata, use the following:
To facilitate S3, GCS, or Azure Data Lake partitioning using a Hive-like partition name format, such as date=yyyy-MM-dd / hour=HH, employ the following SMT configuration for a partition strategy.
and in the KCQL setting utilise the headers as partitioning keys:
Inserts date, year, month, day, hour, minute and second headers using the record timestamp and a rolling time window configuration. If the record timestamp is null, the SMT uses the system time.
The headers inserted are of type STRING. By using this SMT, you can partition the data by yyyy-MM-dd/HH or yyyy/MM/dd/HH, for example, and only use one SMT.
The list of headers inserted are:
date
year
month
day
hour
minute
second
All headers can be prefixed with a custom prefix. For example, if the prefix is wallclock_, then the headers will be:
wallclock_date
wallclock_year
wallclock_month
wallclock_day
wallclock_hour
wallclock_minute
wallclock_second
When used with the Lenses connectors for S3, GCS or Azure data lake, the headers can be used to partition the data. Considering the headers have been prefixed by _, here are a few KCQL examples:
To store the epoch value, use the following configuration:
To prefix the headers with wallclock_, use the following:
To change the date format, use the following:
To use the timezone Asia/Kolkoata, use the following:
To facilitate S3, GCS, or Azure Data Lake partitioning using a Hive-like partition name format, such as date=yyyy-MM-dd / hour=HH, employ the following SMT configuration for a partition strategy.
and in the KCQL setting utilise the headers as partitioning keys:
Inserts the system clock as a message header.
Use SMT if you want to use more than one date time part. This avoids multiple SMTs and is more efficient.
For example, if you want to partition the data by yyyy-MM-dd/HH, then you can use InsertWallclockHeaders which inserts multiple headers: date, year, month, day, hour, minute, second.
Inserts the system clock as a message header, with a value of type STRING. The value can be either a string representation of the date and time epoch value, or a string representation of the date and time in the format specified for example yyyy-MM-dd HH:mm:ss.SSS.
| Name | Description | Type | Default | Valid Values | Importance |
|---|
To store the epoch value, use the following configuration:
To store a string representation of the date and time in the format yyyy-MM-dd HH:mm:ss.SSS, use the following:
To use the timezone Asia/Kolkoata, use the following:
| Name | Description | Type | Default |
|---|---|---|---|
| Name | Description | Type | Default | Importance |
|---|---|---|---|---|
| Name | Description | Type | Default | Importance |
|---|
| Name | Description | Type | Default | Importance |
|---|
field
The field name. If the key is part of the record Key, prefix with _key; otherwise _value. If _value or _key is not used, it defaults to the record Value to resolve the field.
String
format.from.pattern
Optional DateTimeFormatter-compatible format for the timestamp. Used to parse the input if the input is a string. Multiple (fallback) patterns can be added, comma-separated.
String
unix.precision
Optional. The desired Unix precision for the timestamp: seconds, milliseconds, microseconds, or nanoseconds. Used to parse the input if the input is a Long.
String
milliseconds
header.prefix.name
Optional header prefix.
String
date.format
Optional Java date time formatter.
String
yyyy-MM-dd
year.format
Optional Java date time formatter for the year component.
String
yyyy
month.format
Optional Java date time formatter for the month component.
String
MM
day.format
Optional Java date time formatter for the day component.
String
dd
hour.format
Optional Java date time formatter for the hour component.
String
HH
minute.format
Optional Java date time formatter for the minute component.
String
mm
second.format
Optional Java date time formatter for the second component.
String
ss
timezone
Optional. Sets the timezone. It can be any valid Java timezone.
String
UTC
locale
Optional. Sets the locale. It can be any valid Java locale.
String
en
header.prefix.name
Optional header prefix.
String
Low
date.format
Optional Java date time formatter.
String
yyyy-MM-dd
Low
year.format
Optional Java date time formatter for the year component.
String
yyyy
Low
month.format
Optional Java date time formatter for the month component.
String
MM
Low
day.format
Optional Java date time formatter for the day component.
String
dd
Low
hour.format
Optional Java date time formatter for the hour component.
String
HH
Low
minute.format
Optional Java date time formatter for the minute component.
String
mm
Low
second.format
Optional Java date time formatter for the second component.
String
ss
Low
timezone
Optional. Sets the timezone. It can be any valid Java timezone.
String
UTC
Low
locale
Optional. Sets the locale. It can be any valid Java locale.
String
en
Low
| Optional header prefix. | String | Low |
| Optional Java date time formatter. | String | yyyy-MM-dd | Low |
| Optional Java date time formatter for the year component. | String | yyyy | Low |
| Optional Java date time formatter for the month component. | String | MM | Low |
| Optional Java date time formatter for the day component. | String | dd | Low |
| Optional Java date time formatter for the hour component. | String | HH | Low |
| Optional Java date time formatter for the minute component. | String | mm | Low |
| Optional Java date time formatter for the second component. | String | ss | Low |
| Optional. Sets the timezone. It can be any valid Java timezone. | String | UTC | Low |
| Optional. Sets the locale. It can be any valid Java locale. | String | en | Low |
| Sets the window type. It can be fixed or rolling. | String | minutes | hours, minutes, seconds |
| Sets the window size. It can be any positive integer, and depending on the | Int | 15 |
| Optional header prefix. | String | Low |
| Optional Java date time formatter. | String | yyyy-MM-dd | Low |
| Optional Java date time formatter for the year component. | String | yyyy | Low |
| Optional Java date time formatter for the month component. | String | MM | Low |
| Optional Java date time formatter for the day component. | String | dd | Low |
| Optional Java date time formatter for the hour component. | String | HH | Low |
| Optional Java date time formatter for the minute component. | String | mm | Low |
| Optional Java date time formatter for the second component. | String | ss | Low |
| Optional. Sets the timezone. It can be any valid Java timezone. | String | UTC | Low |
| Optional. Sets the locale. It can be any valid Java locale. | String | en | Low |
| Sets the window type. It can be fixed or rolling. | String | minutes | hours, minutes, seconds |
| Sets the window size. It can be any positive integer, and depending on the | Int | 15 |
| The name of the header to insert the timestamp into. | String | High |
| Sets the header value inserted. It can be epoch or string. If string is used, then the 'format' setting is required." | String | format | epoch,format | High |
| Sets the format of the header value inserted if the type was set to string. It can be any valid java date format. | String | High |
| Sets the timezone. It can be any valid java timezone. Overwrite it when | String | UTC | High |
Inserts the datetime as a message header from a value field and a rolling window configuration.
A Kafka Connect Single Message Transform (SMT) that inserts date, year, month,day, hour, minute and second headers using a timestamp field from the record payload and a rolling time window configuration. The timestamp field can be in various valid formats, including long integers, strings, or date objects. The timestamp field can originate from either the record Key or the record Value. When extracting from the record Key, prefix the field with _key.; otherwise, extract from the record Value by default or explicitly using the field without prefixing. For string-formatted fields, specify a format.from.pattern parameter to define the parsing pattern. Long integer fields are assumed to be Unix timestamps; the desired Unix precision can be specified using the unix.precision parameter.
The headers inserted are of type STRING. By using this SMT, you can partition the data by yyyy-MM-dd/HH or yyyy/MM/dd/HH, for example, and only use one SMT.
The list of headers inserted are:
date
year
month
day
hour
minute
second
All headers can be prefixed with a custom prefix. For example, if the prefix is wallclock_, then the headers will be:
wallclock_date
wallclock_year
wallclock_month
wallclock_day
wallclock_hour
wallclock_minute
wallclock_second
When used with the Lenses connectors for S3, GCS or Azure data lake, the headers can be used to partition the data. Considering the headers have been prefixed by _, here are a few KCQL examples:
To store the epoch value, use the following configuration:
To prefix the headers with wallclock_, use the following:
To change the date format, use the following:
To use the timezone Asia/Kolkoata, use the following:
To facilitate S3, GCS, or Azure Data Lake partitioning using a Hive-like partition name format, such as date=yyyy-MM-dd / hour=HH, employ the following SMT configuration for a partition strategy.
and in the KCQL setting utilise the headers as partitioning keys:
format.from.patternConfiguring multiple format.from.pattern items requires careful thought as to ordering and may indicate that your Kafka topics or data processing techniques are not aligning with best practices. Ideally, each topic should have a single, consistent message format to ensure data integrity and simplify processing.
The format.from.pattern field supports multiple DateTimeFormatter patterns in a comma-separated list to handle various timestamp formats. Patterns containing commas should be enclosed in double quotes. For example:
While this flexibility can be useful, it is generally not recommended due to potential complexity and inconsistency. Ideally, a topic should have a single message format to align with Kafka best practices, ensuring consistency and simplifying data processing.
The order of patterns in format.from.pattern matters. Less granular formats should follow more specific ones to avoid data loss. For example, place yyyy-MM-dd after yyyy-MM-dd'T'HH:mm:ss to ensure detailed timestamp information is preserved.
A Kafka Connect Single Message Transform (SMT) that inserts date, year, month,day, hour, minute and second headers using the system clock as a message header.
The headers inserted are of type STRING. By using this SMT, you can partition the data by yyyy-MM-dd/HH or yyyy/MM/dd/HH, for example, and only use one SMT.
The list of headers inserted are:
date
year
month
day
hour
minute
second
All headers can be prefixed with a custom prefix. For example, if the prefix is wallclock_, then the headers will be:
wallclock_date
wallclock_year
wallclock_month
wallclock_day
wallclock_hour
wallclock_minute
wallclock_second
When used with the Lenses connectors for S3, GCS or Azure data lake, the headers can be used to partition the data. Considering the headers have been prefixed by _, here are a few KCQL examples:
To store the epoch value, use the following configuration:
To prefix the headers with wallclock_, use the following:
To change the date format, use the following:
To use the timezone Asia/Kolkoata, use the following:
To facilitate S3, GCS, or Azure Data Lake partitioning using a Hive-like partition name format, such as date=yyyy-MM-dd / hour=HH, employ the following SMT configuration for a partition strategy.
and in the KCQL setting utilise the headers as partitioning keys:
SMT that allows the user to specify the format of the timestamp inserted as a header. It also avoids the synchronization block requirement for converting to a string representation of the timestamp.
An adapted version of the TimestampConverter SMT. The SMT adds a few more features to the original:
allows nested fields resolution (e.g. a.b.c)
uses _key or _value as prefix to understand the field to convert is part of the record Key or Value
allows conversion from one string representation to another (e.g. yyyy-MM-dd HH:mm:ss to yyyy-MM-dd)
allows conversion using a rolling window boundary (e.g. every 15 minutes, or one hour)
| Name | Description | Type | Default | Valid Values |
|---|---|---|---|---|
To convert to and from a string representation of the date and time in the format yyyy-MM-dd HH:mm:ss.SSS, use the following configuration:
To convert to and from a string representation while applying an hourly rolling window:
To convert to and from a string representation while applying an hourly rolling window and timezone:
To convert to and from a string representation while applying a 15 minutes rolling window:
To convert to and from a Unix timestamp, use the following:
Here is an example using the record timestamp field:
format.from.patternConfiguring multiple format.from.pattern items requires careful thought as to ordering and may indicate that your Kafka topics or data processing techniques are not aligning with best practices. Ideally, each topic should have a single, consistent message format to ensure data integrity and simplify processing.
The format.from.pattern field supports multiple DateTimeFormatter patterns in a comma-separated list to handle various timestamp formats. Patterns containing commas should be enclosed in double quotes. For example:
While this flexibility can be useful, it is generally not recommended due to potential complexity and inconsistency. Ideally, a topic should have a single message format to align with Kafka best practices, ensuring consistency and simplifying data processing.
The order of patterns in format.from.pattern matters. Less granular formats should follow more specific ones to avoid data loss. For example, place yyyy-MM-dd after yyyy-MM-dd'T'HH:mm:ss to ensure detailed timestamp information is preserved.
SMT that inserts the system clock value as a message header, a value adapted to a specified time window boundary, for example every 15 minutes, or one hour.
The value inserted is stored as a STRING, and it holds either a string representation of the date and time epoch value, or a string representation of the date and time in the format specified.
| Name | Description | Type | Default | Valid Values | Importance |
|---|---|---|---|---|---|
To store the epoch value, use the following configuration:
To store a string representation of the date and time in the format yyyy-MM-dd HH:mm:ss.SSS, use the following:
To use the timezone Asia/Kolkoata, use the following:
The InsertSourcePartitionOrOffsetValue transformation in Kafka Connect allows you to insert headers into SourceRecords based on partition or offset values.
The InsertSourcePartitionOrOffsetValue transformation in Kafka Connect allows you to insert headers into SourceRecords based on partition or offset values. This is useful for adding metadata to your data records before they are sent to destinations like AWS S3, Azure Datalake, or GCP Storage.
This SMT only works with source connectors.
| Configuration Property | Description | Optionality | Default Value |
|---|---|---|---|
Default Value: Specifies the default value assigned if no value is explicitly provided in the configuration.
These properties allow you to customize which fields from the offset and partition of a SourceRecord are added as headers, along with specifying optional prefixes for the header keys. Adjust these configurations based on your specific use case and data requirements.
transforms: This property lists the transformations to be applied to the records.
transforms.InsertSourcePartitionOrOffsetValue.type: Specifies the class implementing the transformation (InsertSourcePartitionOrOffsetValue in this case).
transforms.InsertSourcePartitionOrOffsetValue.offset.fields: Defines the fields from the offset to be inserted as headers in the SourceRecord. Replace path,line,ts with the actual field names you want to extract from the offset.
transforms.InsertSourcePartitionOrOffsetValue.partition.fields: Defines the fields from the partition to be inserted as headers in the SourceRecord. Replace container,prefix with the actual field names you want to extract from the partition.
When using this transformation with AWS S3, you can configure your Kafka Connect connector as follows:
To customise the header prefix you can also set the header values:
Replace path,line,ts and container,prefix with the actual field names you are interested in extracting from the partition or offset.
By using InsertSourcePartitionOrOffsetValue transformation, you can enrich your data records with additional metadata headers based on partition or offset values before they are delivered to your cloud storage destinations.
The prefix feature in InsertSourcePartitionOrOffsetValue allows you to prepend a consistent identifier to each header key added based on partition or offset values from SourceRecords.
Configure the transformation in your Kafka Connect connector properties:
offset.prefix: Specifies the prefix for headers derived from offset values. Default is "offset.".
partition.prefix: Specifies the prefix for headers derived from partition values. Default is "partition.".
By setting offset.prefix=offset. and partition.prefix=partition., headers added based on offset and partition fields will have keys prefixed accordingly in the SourceRecord headers.
This configuration ensures clarity and organization when inserting metadata headers into your Kafka records, distinguishing them based on their source (offset or partition). Adjust prefixes (offset.prefix and partition.prefix) as per your naming conventions or requirements.
A Kafka Connect Single Message Transform (SMT) that inserts the system clock year, month, day, minute, or seconds as a message header, with a value of type STRING.
Use InsertWallclockHeaders SMT if you want to use more than one date time part. This avoids multiple SMTs and is more efficient.
For example, if you want to partition the data by yyyy-MM-dd/HH, then you can use InsertWallclockHeaders which inserts multiple headers: date, year, month, day, hour, minute, second.
| Name | Description | Type | Default | Valid Values | Importance |
|---|---|---|---|---|---|
To store the year, use the following configuration:
To store the month, use the following configuration:
To store the day, use the following configuration:
To store the hour, use the following configuration:
To store the hour, and apply a timezone, use the following configuration:
To store the minute, use the following configuration:
To store the second, use the following configuration:
| Name | Description | Type | Default |
|---|---|---|---|
| Name | Description | Type | Default | Importance |
|---|
field
The field name. If the key is part of the record Key, prefix with _key; otherwise _value. If _value or _key is not used, it defaults to the record Value to resolve the field.
String
format.from.pattern
Optional DateTimeFormatter-compatible format for the timestamp. Used to parse the input if the input is a string. Multiple (fallback) patterns can be added, comma-separated.
String
unix.precision
Optional. The desired Unix precision for the timestamp: seconds, milliseconds, microseconds, or nanoseconds. Used to parse the input if the input is a Long.
String
milliseconds
header.prefix.name
Optional header prefix.
String
date.format
Optional Java date time formatter.
String
yyyy-MM-dd
year.format
Optional Java date time formatter for the year component.
String
yyyy
month.format
Optional Java date time formatter for the month component.
String
MM
day.format
Optional Java date time formatter for the day component.
String
dd
hour.format
Optional Java date time formatter for the hour component.
String
HH
minute.format
Optional Java date time formatter for the minute component.
String
mm
second.format
Optional Java date time formatter for the second component.
String
ss
timezone
Optional. Sets the timezone. It can be any valid Java timezone.
String
UTC
locale
Optional. Sets the locale. It can be any valid Java locale.
String
en
rolling.window.type
Sets the window type. It can be fixed or rolling.
String
minutes
rolling.window.size
Sets the window size. It can be any positive integer, and depending on the window.type it has an upper bound, 60 for seconds and minutes, and 24 for hours.
Int
15
## Example
header.name
The name of the header to insert the timestamp into.
String
field
The field path containing the timestamp, or empty if the entire value is a timestamp. Prefix the path with the literal string _key, _value or _timestamp, to specify the record Key, Value or Timestamp is used as source. If not specified _value is implied.
String
target.type
Sets the desired timestamp representation.
String
string,unix,date,time,timestamp
format.from.pattern
Sets the format of the timestamp when the input is a string. The format requires a Java DateTimeFormatter-compatible pattern. Multiple (fallback) patterns can be added, comma-separated.
String
format.to.pattern
Sets the format of the timestamp when the output is a string. The format requires a Java DateTimeFormatter-compatible pattern.
String
rolling.window.type
An optional parameter for the rolling time window type. When set it will adjust the output value according to the time window boundary.
String
none
none, hours, minutes, seconds
rolling.window.size
An optional positive integer parameter for the rolling time window size. When rolling.window.type is defined this setting is required. The value is bound by the rolling.window.type configuration. If type is minutes or seconds then the value cannot bigger than 60, and if the type is hours then the max value is 24.
Int
15
unix.precision
The desired Unix precision for the timestamp. Used to generate the output when type=unix or used to parse the input if the input is a Long. This SMT will cause precision loss during conversions from, and to, values with sub-millisecond components.
String
milliseconds
seconds, milliseconds, microseconds, nanoseconds
timezone
Sets the timezone. It can be any valid java timezone. Overwrite it when target.type is set to date, time, or string, otherwise it will raise an exception.
String
UTC
header.name
The name of the header to insert the timestamp into.
String
High
value.type
Sets the header value inserted. It can be epoch or string. If string is used, then the 'format' setting is required."
String
format
epoch,format
High
format
Sets the format of the header value inserted if the type was set to string. It can be any valid java date format.
String
High
rolling.window.type
Sets the window type. It can be fixed or rolling.
String
minutes
hours, minutes, seconds
High
rolling.window.size
Sets the window size. It can be any positive integer, and depending on the window.type it has an upper bound, 60 for seconds and minutes, and 24 for hours.
Int
15
High
timezone
Sets the timezone. It can be any valid java timezone. Overwrite it when value.type is set to format, otherwise it will raise an exception.
String
UTC
High
offset.fields
Comma-separated list of fields to retrieve from the offset
Optional
Empty list
offset.prefix
Optional prefix for offset keys
Optional
"offset."
partition.fields
Comma-separated list of fields to retrieve from the partition
Required
Empty list
partition.prefix
Optional prefix for partition keys
Optional
"partition."
header.name
The name of the header to insert the timestamp into.
String
High
date.time.part
The date time part to insert.
String
year, month, day, hour,minute, second
High
timezone
Sets the timezone. It can be any valid java timezone.
String
UTC
High
| Optional header prefix. | String | Low |
| Optional Java date time formatter. | String | yyyy-MM-dd | Low |
| Optional Java date time formatter for the year component. | String | yyyy | Low |
| Optional Java date time formatter for the month component. | String | MM | Low |
| Optional Java date time formatter for the day component. | String | dd | Low |
| Optional Java date time formatter for the hour component. | String | HH | Low |
| Optional Java date time formatter for the minute component. | String | mm | Low |
| Optional Java date time formatter for the second component. | String | ss | Low |
| Optional. Sets the timezone. It can be any valid Java timezone. | String | UTC | Low |
| Optional. Sets the locale. It can be any valid Java locale. | String | en | Low |
