> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/k2k/configuration-reference/offset-mapping.md).

# Offset mapping

Translate consumer group positions to the target cluster.

{% hint style="info" %}
This feature is **disabled by default**. Set the corresponding `features` toggle to `enabled` in your pipeline file for the behavior described on this page to take effect.
{% endhint %}

{% hint style="warning" %}
**This feature requires a license.** It is unavailable in free tier mode, and the license token must carry the matching entitlement. Enabling it with a token that does not cover it fails the pipeline at startup with a license restriction error.

See [Licensed features](https://github.com/lensesio-dev/k2k-documentation/tree/release/2.1.0/.gitbook/includes/licenses.md#licensed-features).
{% endhint %}

{% hint style="warning" %}
When `features.offsetMapping` is `enabled`, deploy and run both `k2k` and `k2k-offset-mapping`. Enabling the feature in the pipeline file does not translate consumer offsets by itself.
{% endhint %}

Kafka stores consumer positions relative to each cluster's partition log. The offset where an application stopped on the the source Kafka cluster has no meaning on the target Kafka cluster — therefore failover would force consumers to reprocess from the beginning unless we translate positions.

K2K records source-target-Kafka-cluster offset mappings as it replicates, then the offset mapper translates committed consumer group positions on the target Kafka cluster. Both applications share the same pipeline file; see [Deployment](/latest/k2k/deployment.md). Rather than reading `__consumer_offsets` directly, K2K polls group metadata through the admin API, which supports managed offerings where internal topics are not exposed. For topic provisioning, see [Topic creation](/latest/k2k/configuration-reference/topic-creation.md). For polling, retention, and error handling, see the [Advanced](#advanced) section below.

## Deployment

Offset mapping requires **both** K2K applications running with the same pipeline file:

| Application       | Command                    | Role                                                            |
| ----------------- | -------------------------- | --------------------------------------------------------------- |
| K2K replicator    | `k2k start`                | Records source-to-target offset mappings as it replicates       |
| K2K offset mapper | `k2k-offset-mapping start` | Translates consumer group positions on the target Kafka cluster |

```shell
k2k start -f my_pipeline.yml
k2k-offset-mapping start -f my_pipeline.yml
```

Keep both processes running continuously. Enabling `features.offsetMapping` in the pipeline file alone does not translate consumer offsets. See [Deployment](/latest/k2k/deployment.md#offset-mapping-both-applications-required).

## Prerequisites

* [License accepted](/latest/k2k/configuration-reference/licenses.md), with a token carrying the offset mapping entitlement
* [Kafka connectivity and authentication](/latest/k2k/configuration-reference/kafka-connectivity-authentication.md)
* [Topic selection and data routing](/latest/k2k/configuration-reference/topic-selection-and-data-routing.md) configured with `partition: source`
* `target.kafka.consumer."group.id"` set — the offset mapper joins a consumer group on the target Kafka cluster to read the mapping topic
* `features.offsetMapping: enabled` in the pipeline
* Replicator running (`k2k start`) with the pipeline above
* Offset mapper running (`k2k-offset-mapping start`) with the same pipeline file

## Example

```yaml
name: "my_pipeline"
features:
  offsetMapping: enabled
source:
  kafka:
    common:
      "bootstrap.servers": "source:9092"
    consumer:
      "group.id": "my-k2k-group"
target:
  kafka:
    common:
      "bootstrap.servers": "target:9092"
    consumer:
      "group.id": "my-k2k-offset-mapper"
groups:
  consumerGroups: ".*"
replication:
  - source:
      topic: "orders.*"
  - sink:
      topic: source
      partition: source
```

`target.kafka.consumer."group.id"` is what the offset mapper uses to join a consumer group on the target Kafka cluster and read the mapping topic. It is unrelated to `source.kafka.consumer."group.id"`, which is the [K2K group](/latest/k2k/configuration-reference/consumer-group.md) the replicator uses. All offset mapper instances for a pipeline must share it — see [Horizontal scaling](/latest/k2k/configuration-reference/horizontal-scaling.md#offset-mapper).

Start both applications from the same pipeline file:

```shell
k2k start -f my_pipeline.yml
k2k-offset-mapping start -f my_pipeline.yml
```

## Configuration keys

### Feature toggle

| Key                      | Description                                                              | Required | Default    | Used by    |
| ------------------------ | ------------------------------------------------------------------------ | -------- | ---------- | ---------- |
| `features.offsetMapping` | Record offset mappings during replication for consumer group translation | No       | `disabled` | Replicator |

Both the replicator and the offset mapper check the offset mapping entitlement, so a token without it fails both applications at startup.

### Connection keys

| Key                                | Description                                                              | Required    | Default | Used by       |
| ---------------------------------- | ------------------------------------------------------------------------ | ----------- | ------- | ------------- |
| `target.kafka.consumer."group.id"` | Consumer group the offset mapper joins to read the offset mapping topic  | Yes         | (none)  | Offset mapper |
| `groups.consumerGroups`            | Consumer groups on the source Kafka cluster to translate (regex or list) | Conditional | `.*`    | Offset mapper |

## Advanced

### Offset mapping topic

| Key                   | Description                                                                                            | Required | Default                 | Used by |
| --------------------- | ------------------------------------------------------------------------------------------------------ | -------- | ----------------------- | ------- |
| `offsetMapping.topic` | Topic storing source-Kafka-cluster-to-target-Kafka-cluster offset mappings on the target Kafka cluster | No       | `__k2k_offset_mappings` | Both    |

For auto creating this topic with custom settings, see [Topic creation](/latest/k2k/configuration-reference/topic-creation.md).

### Mappings guard

| Key                           | Description                                                                                            | Required | Default   | Used by       |
| ----------------------------- | ------------------------------------------------------------------------------------------------------ | -------- | --------- | ------------- |
| `features.applyMappingsGuard` | Skip applying offset mappings when the target Kafka cluster already has a more recent committed offset | No       | `enabled` | Offset mapper |

The guard is **on by default**. The offset mapper checks committed offsets on the target Kafka cluster before applying a translation; if the target group already has a newer position, the mapping is skipped to avoid moving consumers backwards. Set it to `disabled` only when you want translations applied unconditionally.

### Polling and group selection

| Key                     | Description                                                                                    | Required    | Default     | Used by       |
| ----------------------- | ---------------------------------------------------------------------------------------------- | ----------- | ----------- | ------------- |
| `groups.consumerGroups` | Consumer groups on the source Kafka cluster to translate (regex string or list of group names) | Conditional | `.*`        | Offset mapper |
| `groups.pollInterval`   | How often to poll the source Kafka cluster for consumer group changes                          | No          | `5 seconds` | Offset mapper |

On each poll, K2K lists consumer groups on the source Kafka cluster and fetches their committed offsets through the admin API. This avoids depending on direct access to internal `__consumer_offsets` topics, which some managed Kafka vendors do not expose.

`groups.consumerGroups` accepts two forms. A **regex string** matches group names by pattern:

```yaml
groups:
  consumerGroups: "orders-.*"
```

A **YAML list** names groups exactly, which is the safer choice for a controlled failover where only specific applications should be translated:

```yaml
groups:
  consumerGroups:
    - "orders-service"
    - "payments-service"
```

### Retention

| Key                                | Description                                                            | Required | Default     | Used by       |
| ---------------------------------- | ---------------------------------------------------------------------- | -------- | ----------- | ------------- |
| `offsetMapping.retention.time`     | How long to retain mapping ranges (`topic`, `infinite`, or a duration) | No       | `topic`     | Offset mapper |
| `offsetMapping.retention.count`    | Mapping ranges to retain per topic partition (integer or `infinite`)   | No       | `1000`      | Offset mapper |
| `offsetMapping.retention.schedule` | How often the retention process runs                                   | No       | `5 minutes` | Offset mapper |

Retention controls how many historical cross-cluster offset ranges K2K keeps on the target Kafka cluster. Older ranges are removed so the mapping topic does not grow without bound.

`retention.time` accepts `topic` (follow the mapping topic's own retention), `infinite` (never evict by age), or a duration such as `7 days`. `retention.count` accepts an integer or `infinite` (never evict by count). Both limits apply together — a range is evicted when it exceeds either one.

### Watermark prefix

| Key                        | Description                                                                         | Required | Default | Used by       |
| -------------------------- | ----------------------------------------------------------------------------------- | -------- | ------- | ------------- |
| `groups.offsetGuardPrefix` | Prefix for watermark entries written to consumer groups on the target Kafka cluster | No       | `k2k_`  | Offset mapper |

K2K writes watermark offsets to the target Kafka cluster under synthetic topic names prefixed with this value. The offset mapper uses these watermarks to detect when a target consumer group already has a more recent position than a pending translation.

### Error handling

| Key                                              | Description                                                                                  | Required | Default | Used by       |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------- | -------- | ------- | ------------- |
| `errorHandling.onMultipleTargetCommittedOffsets` | Action when multiple committed offsets exist for the same group on the target Kafka cluster  | No       | `fail`  | Offset mapper |
| `errorHandling.onActiveConsumerGroupCommit`      | Action when committing offsets for a group with active consumers on the target Kafka cluster | No       | `fail`  | Offset mapper |

`onMultipleTargetCommittedOffsets` applies when the target Kafka cluster reports more than one committed offset for the same group and partition. `onActiveConsumerGroupCommit` applies when the offset mapper would commit offsets for a group that currently has active consumers on the target Kafka cluster.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.lenses.io/latest/k2k/configuration-reference/offset-mapping.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.