Configuration Reference

In order for the consumer offset mapping functionality to work, both of the following requirements must be met:

• K2K must be configured to publish offset mappings.

• The k2k-offset-mapping application must be configured and deployed.

Without both components active, consumer group offsets on the target cluster will not be translated.

Feature Flags

Field	Behavior/Description	Required	Default	Type
features.autoCreateControlTopics	Enables creation of the necessary control topics used by the k2k application.		disabled	disabled, enabled
features.applyMappingsGuards	Enables/Disables the behavior described in Offset Rewind Protection		enabled	disabled, enabled

Pipeline

Field	Behavior/Description	Required	Default	Type
name	name of the offset-mapping app			string

License

Field	Behavior/Description	Required	Default	Type
license.acceptEula	It needs to be set to true to run the K2K application			boolean
license.token	A license key to enable premium features like exactly-once semantics [email protected]			string

Error Handling

Field	Behavior/Description	Required	Default	Type
errorHandling.onMultipleTargetCommittedOffsets	Handles error behavior when multiple options exist for mapping of a record.		fail	fail, ignore
errorHandling.onActiveConsumerGroupCommit	Handles error behavior when a commit is attempted for a group with an active consumer.		fail	fail, ignore

Group configuration

Field	Behavior/Description	Required	Default	Type
groups.pollInterval	Interval between calls for consumer group details.		5 seconds	duration string
groups.consumerGroups	Filter that selects which consumer groups should be synced to the target cluster. Refer to Group Filtering for the detailed list of possibilities.		.*	list of string,regex
groups.offsetGuardPrefix	Prefix added to consumer group metadata offset guard.Refer to Offset Rewind Protection for more details		k2k_	string

Offset Mapping

Field	Behavior/Description	Required	Default	Type
offsetMapping.topic	Control topic used to read offset mapping information.		__k2k_offset_mappings	string

Source Cluster Kafka

Only the connection details are required in this section (bootstrap servers and credentials). Those can be provided in the common section and will be inherited by the other Kafka client configurations targeting this cluster.

Field	Behavior/Description	Required	Default	Type
source.kafka.common	Common properties inherited by consumer and admin clients			object
source.kafka.admin	Common properties inherited by consumer and admin and producer clients.			object

Target Cluster Kafka and Schema Registry Connection

Only the connection details are required in this section (bootstrap servers and credentials). Those can be provided in the common section and will be inherited by the other Kafka client configurations targeting this cluster.

Field	Behavior/Description	Required	Default	Type
target.kafka.common	Common properties inherited by consumer, admin and producer clients			object
target.kafka.producer	Kafka producer configuration used to produce metrics to the target cluster (if enabled)			object
target.kafka.admin	Kafka admin client configuration used to introspect the Kafka target cluster.			object
target.kafka.consumer	Kafka consumer configuration used to read offset mappings present in the Kafka target cluster.

Metrics

Field	Behavior/Description	Required	Default	Type
metrics.prefix.targetConsumer	Prefix added to published metrics about the K2K Offset Mapping target consumer.		k2k.offsetmapping.consumer.control.	cluster , pipeline
metrics.prefix.targetProducer	Prefix added to published metrics about the K2K Offset Mapping target producer.		k2k.offsetmapping.producer.	enabled
metrics.prefix.offsetMappingApp	Prefix added to published metrics about the K2K Offset Mapping app.		k2k.offsetmapping.	enabled

Control Topics Auto Creation

Only relevant if features.autoCreateTopics is enabled.

Field	Behavior / Description	Required	Default	Type
topicCreation.control.common.partitions	Default partition count to be used when auto-creating any of the replicated topics.			integer
topicCreation.control.common.replication	Default replication to be used when auto-creating any of the replicated control topics.			short
topicCreation.control.common.config	Default topic configuration to be used when auto-creating any of the replicated control topics.		empty object	object
topicCreation.control.metrics.partitions	Default partition count to be used when creating the topic used for metrics.			integer
topicCreation.control.metrics.replication	Default replication to be used when creating the topic used for metrics.			integer
topicCreation.control.metrics.config	Default topic configuration to be used when creating the topic used for metrics.			object
topicCreation.control.offsetMapping.partitions	Default partition count to be used when auto-creating the offset-mappings topic.			integer
topicCreation.control.offsetMapping.replication	Default replication to be used when auto-creating the offset-mappings topic.			integer
topicCreation.control.offsetMapping.config	Default config to be used when auto-creating the offset-mappings topic.		object with "cleanup.policy": "delete"	object

• If any of the properties topicCreation.*.replication and topicCreation.*.partitions is left blank (or set to null), K2K will default to the cluster default values for those properties.or control topics, the final topic configuration will be the result of merging the default/common value with the one specified in topicCreation.control.common.config and the one for the specific topic e.g: topicCreation.control.commit.config.

• For replicated topics, the final topic configuration will be the result of overriding the original topic's configuration with the result of merging the default/common value specified in topicCreation.replication.common with the one specified in the first matching rule in topicCreation.replication.rules[].