mapConsumer Offset Mapping

circle-exclamation

When replicating records between Kafka clusters, it is not possible to guarantee that a record's offset on the source cluster will be the same as its offset on the target cluster. This discrepancy is inherent to cross-cluster replication: retention policies, Kafka transactions or prior data on the target can all cause offsets to diverge.

To address this constraint, K2K can be configured to publish offset mappings alongside the replicated data. When this feature is enabled, K2K will also publish a mapping that correlates the record's original source offset with its new offset on the target cluster.

This mapping data is published to a dedicated topic on the target cluster. A separate application is then required to read these mappings and continuously translate consumer group offsets between the source and target clusters. This enables consumer groups connecting to the target Kafka cluster to resume from the correct position relative to the source, even when the underlying offsets do not match.

Last updated

Was this helpful?