> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/k2k/configuration-reference/topic-selection-and-data-routing.md).

# Topic selection and data routing

Which topics to replicate and how records are routed.

A replication pipeline must answer two questions: which topics to read on the source Kafka cluster, and where each record lands on the target Kafka cluster. Kafka has no built-in concept of cross-cluster routing, so K2K expresses this through a source stage and a sink stage in the pipeline.

Each pipeline has exactly one source stage (topic selection) and one sink stage (topic and partition assignment). See [Replication stage](https://docs.lenses.io/latest/k2k/glossary.md#replication) in the Glossary. For header replication, see [Header replication](https://docs.lenses.io/latest/k2k/configuration-reference/header-replication.md). If you use [offset mapping](https://docs.lenses.io/latest/k2k/configuration-reference/offset-mapping.md), set `partition: source` so partition N on the source Kafka cluster maps to partition N on the target Kafka cluster

## Prerequisites

* [License accepted](https://docs.lenses.io/latest/k2k/configuration-reference/licenses.md)
* [Kafka connectivity and authentication](https://docs.lenses.io/latest/k2k/configuration-reference/kafka-connectivity-authentication.md)
* [Consumer group](https://docs.lenses.io/latest/k2k/configuration-reference/consumer-group.md) configured
* One source stage and one sink stage in `replication`

## Example

```yaml
name: "my_pipeline"
source:
  kafka:
    common:
      "bootstrap.servers": "source:9092"
    consumer:
      "group.id": "my-k2k-group"
target:
  kafka:
    common:
      "bootstrap.servers": "target:9092"
replication:
  - source:
      topic: "orders.*"
  - sink:
      topic:
        prefix: "dr-"
      partition: source
```

## Configuration keys

### Source stage

| Key                          | Description                                               | Required | Default | Used by    |
| ---------------------------- | --------------------------------------------------------- | -------- | ------- | ---------- |
| `replication[].source.topic` | Topics to replicate (regex string or list of topic names) | Yes      | (none)  | Replicator |

### Sink stage

| Key                            | Description                          | Required | Default | Used by    |
| ------------------------------ | ------------------------------------ | -------- | ------- | ---------- |
| `replication[].sink.topic`     | Target topic assignment strategy     | Yes      | (none)  | Replicator |
| `replication[].sink.partition` | Target partition assignment strategy | Yes      | (none)  | Replicator |

Both keys must be set explicitly — there is no implied default. Use `source` for either to preserve the source value.

## Source topic selection

**Regex.** Provide a string to match topic names on the source Kafka cluster:

```yaml
- source:
    topic: "orders.*"
```

**Topic list.** Provide a YAML list of explicit topic names:

```yaml
- source:
    topic:
      - "orders"
      - "payments"
      - "audit-log"
```

## Data routing

The sink stage controls how each record read from the source Kafka cluster is written to the target Kafka cluster.

By default (`topic: source`, `partition: source`), a record from topic `orders` on partition `3` of the source Kafka cluster is written to topic `orders` on partition `3` of the target Kafka cluster. The topic name and partition number are preserved.

Other strategies rename topics, consolidate multiple source topics into one target topic, or change how partitions are assigned.

### Sink topic strategies

**Same name as source** (`topic: source`):

```yaml
- sink:
    topic: source
    partition: source
```

Records keep the same topic name on the target Kafka cluster.

**Static topic** (all records to one topic):

```yaml
- sink:
    topic:
      static: "target-topic-name"
    partition: source
```

All replicated records are written to `target-topic-name` on the target Kafka cluster, regardless of their source topic.

**Prefix and/or suffix** (transform source topic name):

```yaml
- sink:
    topic:
      prefix: "prod-"
      suffix: "-replica"
    partition: source
```

A record from topic `orders` on the source Kafka cluster is written to `prod-orders-replica` on the target Kafka cluster.

**Static mapping** (route by source topic pattern). This controls routing only; which topics are replicated is defined by the source stage:

```yaml
- sink:
    topic:
      staticMapping:
        team-red.*: "team-red-data"
        team-blue.*: "team-blue-data"
        .*: "other-teams-data"
    partition: source
```

### Sink partition strategies

| Value      | Behaviour                                                                   |
| ---------- | --------------------------------------------------------------------------- |
| `source`   | Same partition number as on the source Kafka cluster                        |
| `producer` | Partition chosen by the Kafka producer strategy on the target Kafka cluster |

```yaml
- sink:
    topic: source
    partition: producer
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.lenses.io/latest/k2k/configuration-reference/topic-selection-and-data-routing.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
