K2K Offset Mapping

In order for the consumer offset mapping functionality to work, both of the following requirements must be met:

• K2K must be configured to publish offset mappings.

• The k2k-offset-mapping application must be configured and deployed.

Without both components active, consumer group offsets on the target cluster will not be translated.

When replicating records between Kafka clusters, it is not possible to guarantee that a record's offset on the source cluster will be the same as its offset on the target cluster. This discrepancy is inherent to cross-cluster replication: retention policies, Kafka transactions or prior data on the target can all cause offsets to diverge.

To address this constraint, K2K can be configured to publish offset mappings alongside the replicated data. When this feature is enabled, K2K will also publish a mapping that correlates the record's original source offset with its new offset on the target cluster.

This mapping data is published to a dedicated topic on the target cluster.

The K2K Consumer Offset Mapping App is a separate application is then required to read these mappings and continuously sync and translate consumer group offsets between the source and target clusters. This enables consumer groups connecting to the target Kafka cluster to resume from the correct position relative to the source, even when the underlying offsets do not match.

Bellow is the full set of configuration options available for the K2K Offset Mapping application. Before copying it however, do read the Reusing the K2K configuration .

name: "offset-mapping-production"
license:
  token: "<your-license-token>"
  acceptEula: true
features:
  autoCreateControlTopics: enabled
  applyMappingsGuard: enabled
source:
  kafka:
    common:
      "bootstrap.servers": "source-kafka:9092"
    admin:
target:
  kafka:
    common:
      "bootstrap.servers": "target-kafka:9093"
    consumer:
    producer:
    admin:
groups:
  pollInterval: "5 seconds"
  consumerGroups: "^my-app-.*"
  offsetGuardPrefix: "k2k_"
offsetMapping:
  topic: "__k2k_offset_mappings"
  retention:
    time: "topic"
    count: 1000
    schedule: "5 minutes"
metrics:
  prefix:
    targetConsumer: "k2k.offsetmapping.consumer.control."
    targetProducer: "k2k.offsetmapping.producer."
    offsetMappingApp: "k2k.offsetmapping."
  kafka:
    topic: "__k2k_metrics"
topicCreation:
  control:
    common:
      partitions: 3
      replication: 3
      config:
        "min.insync.replicas": "2"
    offsetMapping:
      partitions: 6
      replication: 3
      config:
        "cleanup.policy": "delete"
        "retention.ms": "604800000"
    metrics:
      partitions: 3
      replication: 3
      config:
        "cleanup.policy": "delete"
        "retention.ms": "2592000000"
configuration:
  autoControlTopicCreationCheckDelay: "750 milliseconds"
  autoControlTopicCreationMaxAttempts: 10
errorHandling:
  onMultipleTargetCommittedOffsets: fail
  onActiveConsumerGroupCommit: fail