> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/k2k/configuration-reference/horizontal-scaling.md).

# Horizontal scaling

Run multiple instances for higher throughput.

Kafka distributes partitions across consumer group members — one instance cannot read faster than its assigned partitions allow. When production on the source Kafka cluster outpaces a single replicator, therefore, we run multiple instances that share the same pipeline and consumer group.

For the replicator, Kafka assigns source topic partitions across instances in the [K2K group](/latest/k2k/configuration-reference/consumer-group.md). Kafka caps how many instances can do useful work: extra instances sit idle when every partition is already assigned. Depending on the consumer `partition.assignment.strategy`, the cap is either the highest partition count among subscribed source topics (range assignor) or the total partition count across those topics (round-robin and cooperative sticky assignors).

The offset mapper scales differently: Kafka assigns partitions of the internal offset mapping topic (`offsetMapping.topic`, default `__k2k_offset_mappings`) across offset mapper instances in their target Kafka cluster consumer group. Increase that topic's partition count to add parallel capacity — see [Topic creation](/latest/k2k/configuration-reference/topic-creation.md#control-topics).

See [Exactly once delivery](/latest/k2k/configuration-reference/exactly-once-delivery.md) and [Commit tuning](/latest/k2k/configuration-reference/commit-tuning.md). For per-instance tuning, see the [Advanced](#advanced) section below.

## Prerequisites

* [License accepted](/latest/k2k/configuration-reference/licenses.md)
* [Kafka connectivity and authentication](/latest/k2k/configuration-reference/kafka-connectivity-authentication.md)
* [Consumer group](/latest/k2k/configuration-reference/consumer-group.md) configured
* [Topic selection and data routing](/latest/k2k/configuration-reference/topic-selection-and-data-routing.md) configured
* For exactly once setups, [exactly once delivery](/latest/k2k/configuration-reference/exactly-once-delivery.md) configured

## How it works

### Replicator

Start multiple replicator processes with the same pipeline configuration. Each instance uses the same `source.kafka.consumer."group.id"` and joins the same [K2K group](/latest/k2k/glossary.md#consumer-groups).

Kafka assigns each source topic partition to one group member. Each instance reads from the partitions assigned to it. Adding instances beyond the partition cap configured by the assignor does not increase throughput.

### Offset mapper <a href="#offset-mapper" id="offset-mapper"></a>

Start multiple offset mapper processes with the same pipeline configuration. Each instance joins the target Kafka cluster consumer group (`target.kafka.consumer."group.id"` — required for the offset mapper, and shared by every instance of the pipeline) and reads from partitions of the offset mapping topic assigned to it. Offset mapper instances load mappings from their assigned partitions and apply translations for all monitored consumer groups. Parallel capacity is therefore bounded by the partition count of `offsetMapping.topic` on the target Kafka cluster, not by source topic partitions. Configure it under `topicCreation.control.offsetMapping` or `topicCreation.control.common` — see [Topic creation](/latest/k2k/configuration-reference/topic-creation.md).

## Example

Use the same pipeline file on every instance:

```shell
k2k start -f my_pipeline.yml
```

```yaml
name: "my_pipeline"
source:
  kafka:
    common:
      "bootstrap.servers": "source:9092"
    consumer:
      "group.id": "my-k2k-group"
target:
  kafka:
    common:
      "bootstrap.servers": "target:9092"
replication:
  - source:
      topic: "orders.*"
  - sink:
      topic: source
      partition: source
```

Deploy one process per host, container, or pod. No additional configuration is required beyond a standard pipeline definition.

## Advanced

### Per-instance throughput

Higher `pipelineParallelism` increases concurrent partition processing but uses more memory. `maxPrefetchBatches` controls how far ahead each partition reads while earlier batches are still processing. For consumer, producer, and commit tuning, see [Replication performance tuning](/latest/k2k/configuration-reference/replication-performance-tuning.md).

| Key                                 | Description                                                                                         | Required | Default          | Used by    |
| ----------------------------------- | --------------------------------------------------------------------------------------------------- | -------- | ---------------- | ---------- |
| `configuration.pipelineParallelism` | Max concurrent streams processing assigned partitions                                               | No       | `15`             | Replicator |
| `configuration.maxPrefetchBatches`  | Record batches prefetched per partition while earlier batches process (minimum effective value `2`) | No       | `3`              | Replicator |
| `configuration.pollInterval`        | Minimum interval between Kafka consumer poll calls                                                  | No       | `0 milliseconds` | Replicator |
| `configuration.controlParallelism`  | Max concurrent streams processing control messages                                                  | No       | `10`             | Replicator |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.lenses.io/latest/k2k/configuration-reference/horizontal-scaling.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.