Consumer Offset Mapping

In order for the consumer offset mapping functionality to work, both of the following requirements must be met:

• K2K must be configured to publish offset mappings.

• The k2k-offset-mapping application must be configured and deployed.

Without both components active, consumer group offsets on the target cluster will not be translated.

When replicating records between Kafka clusters, it is not possible to guarantee that a record's offset on the source cluster will be the same as its offset on the target cluster. This discrepancy is inherent to cross-cluster replication: retention policies, Kafka transactions or prior data on the target can all cause offsets to diverge.

To address this constraint, K2K can be configured to publish offset mappings alongside the replicated data. When this feature is enabled, K2K will also publish a mapping that correlates the record's original source offset with its new offset on the target cluster.

This mapping data is published to a dedicated topic on the target cluster. A separate application is then required to read these mappings and continuously translate consumer group offsets between the source and target clusters. This enables consumer groups connecting to the target Kafka cluster to resume from the correct position relative to the source, even when the underlying offsets do not match.