> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/k2k/configuration-reference/replication-performance-tuning.md).

# Replication performance tuning

Consumer, producer, and commit settings for throughput.

Replication throughput is bounded by how fast K2K can read from the source Kafka cluster, process records, write to the target Kafka cluster, and record progress. Each stage buffers data — consumer polls, in-flight producer batches, commit queues — so tuning is always a trade-off between throughput, latency, memory, and CPU.

K2K exposes tuning at three levels: standard Kafka consumer and producer properties passed through the pipeline, pipeline concurrency settings that control how many partitions and batches are processed in parallel, and commit batching that controls how often progress is recorded. For adding instances, see [Horizontal scaling](/latest/k2k/configuration-reference/horizontal-scaling.md). For commit batch semantics in exactly once setups, see [Commit tuning](/latest/k2k/configuration-reference/commit-tuning.md). For Kafka client inheritance, see [Kafka connectivity and authentication](/latest/k2k/configuration-reference/kafka-connectivity-authentication.md).

## Prerequisites

* [License accepted](/latest/k2k/configuration-reference/licenses.md)
* [Kafka connectivity and authentication](/latest/k2k/configuration-reference/kafka-connectivity-authentication.md)
* [Topic selection and data routing](/latest/k2k/configuration-reference/topic-selection-and-data-routing.md) configured

## Example

```yaml
name: "my_pipeline"
source:
  kafka:
    common:
      "bootstrap.servers": "source:9092"
    consumer:
      "group.id": "my-k2k-group"
      "max.poll.records": "2000"
      "fetch.max.bytes": "52428800"
target:
  kafka:
    common:
      "bootstrap.servers": "target:9092"
    producer:
      "batch.size": "524288"
      "linger.ms": "20"
      "buffer.memory": "67108864"
      "compression.type": "lz4"
configuration:
  pipelineParallelism: 15
  maxPrefetchBatches: 3
  pollInterval: "0 milliseconds"
coordination:
  commit:
    batchSize: 30000
    batchTimeout: "30 seconds"
    pendingAckQueueSize: 5
replication:
  - source:
      topic: "orders.*"
  - sink:
      topic: source
      partition: source
```

## How tuning affects resources

| Mechanism                                               | Raises throughput when…                                             | Memory impact                                                         | CPU impact                                 |
| ------------------------------------------------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------ |
| Larger consumer polls (`max.poll.records`, fetch sizes) | Brokers return more data per request                                | Higher — each poll holds more records in the pipeline                 | Lower — fewer fetch round-trips per record |
| Higher `pipelineParallelism`                            | More assigned partitions process records concurrently               | Higher — more partitions active at once                               | Higher                                     |
| Higher `maxPrefetchBatches`                             | The consumer reads ahead while earlier batches are still processing | Higher — more poll batches buffered per partition                     | Moderate                                   |
| Larger producer batches (`batch.size`, `linger.ms`)     | The producer sends fewer, larger requests                           | Higher — `buffer.memory` holds more in-flight data                    | Lower per record; compression adds CPU     |
| Larger commit batches (`coordination.commit.batchSize`) | Progress is recorded less often                                     | Higher — more records wait for acknowledgement before offsets advance | Lower commit overhead                      |
| Shorter `configuration.pollInterval`                    | The consumer polls more frequently                                  | Lower latency to pick up new data                                     | Higher — more poll calls                   |

Start from K2K defaults and change one setting at a time. If heap usage grows with backlog depth, reduce commit batch size, prefetch depth, or producer buffer settings before adding memory.

## Source consumer

The source consumer reads replicated topics. Set standard Kafka consumer properties under `source.kafka.consumer` (or `source.kafka.common` when the same value applies to all source clients).

| Key                                            | Description                                                  | Effect on performance                                                                                                                                        |
| ---------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `source.kafka.consumer."max.poll.records"`     | Maximum records returned per poll                            | Larger values increase per-poll memory and processing units. A commit batch may exceed `coordination.commit.batchSize` by up to one poll's worth of records. |
| `source.kafka.consumer."fetch.max.bytes"`      | Maximum bytes fetched per partition per request              | Raises the per-fetch data ceiling. Useful when records are large.                                                                                            |
| `source.kafka.consumer."fetch.min.bytes"`      | Minimum bytes the broker should accumulate before responding | Higher values improve batching on the broker side at the cost of latency.                                                                                    |
| `source.kafka.consumer."fetch.max.wait.ms"`    | Maximum time the broker waits to fill `fetch.min.bytes`      | Works with `fetch.min.bytes` to trade latency for larger fetches.                                                                                            |
| `source.kafka.consumer."receive.buffer.bytes"` | TCP receive buffer size                                      | Increase on high-throughput or high-latency links.                                                                                                           |

## Target producer

The target producer writes replicated records. Set standard Kafka producer properties under `target.kafka.producer` (or `target.kafka.common`).

| Key                                                             | Description                                                     | Effect on performance                                                                                              |
| --------------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `target.kafka.producer."batch.size"`                            | Target batch size in bytes before sending                       | Larger batches improve throughput and reduce request count. Each partition maintains its own batch.                |
| `target.kafka.producer."linger.ms"`                             | Time to wait for more records before sending a batch            | Non-zero values let batches fill, improving throughput at the cost of latency.                                     |
| `target.kafka.producer."buffer.memory"`                         | Total memory available to the producer for buffering            | When exhausted, `send` blocks and replication stalls. Raise with care — this is a hard memory ceiling.             |
| `target.kafka.producer."compression.type"`                      | Compression algorithm (`none`, `gzip`, `lz4`, `zstd`, `snappy`) | Reduces network and broker load; adds CPU on the K2K host.                                                         |
| `target.kafka.producer."max.in.flight.requests.per.connection"` | Unacknowledged produce requests per connection                  | Higher values improve throughput in at-least-once mode. With exactly once enabled, idempotence limits this to `5`. |

K2K sets `enable.idempotence: true` and `acks: all` on the target producer by default. See [Kafka connectivity and authentication](/latest/k2k/configuration-reference/kafka-connectivity-authentication.md#client-defaults).

## Pipeline concurrency

These K2K settings control how the source consumer and control consumer schedule work inside a single replicator instance. They apply on top of Kafka client properties.

| Key                                 | Description                                                                                                               | Required | Default          | Used by    |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------- | ---------- |
| `configuration.pipelineParallelism` | Max concurrent streams processing assigned source partitions                                                              | No       | `15`             | Replicator |
| `configuration.maxPrefetchBatches`  | Record batches the source consumer may prefetch per partition while earlier batches process (minimum effective value `2`) | No       | `3`              | Replicator |
| `configuration.pollInterval`        | Minimum interval between Kafka consumer poll calls                                                                        | No       | `0 milliseconds` | Replicator |
| `configuration.controlParallelism`  | Max concurrent streams processing control messages                                                                        | No       | `10`             | Replicator |

`pipelineParallelism` should be aligned with the number of partitions assigned to the instance — see [Horizontal scaling](/latest/k2k/configuration-reference/horizontal-scaling.md). `maxPrefetchBatches` multiplies the memory held per partition: each prefetched batch is up to `max.poll.records` records deep.

## Commit frequency

K2K records replication progress in batches rather than after every record. Commit frequency affects both throughput and peak memory — records stay in the pipeline until their batch is acknowledged.

| Key                                        | Description                                                     | Required | Default      | Used by    |
| ------------------------------------------ | --------------------------------------------------------------- | -------- | ------------ | ---------- |
| `coordination.commit.batchSize`            | Records to accumulate before committing progress                | No       | `30000`      | Replicator |
| `coordination.commit.batchTimeout`         | Max wait before committing a partial batch                      | No       | `30 seconds` | Replicator |
| `coordination.commit.parallelizationLimit` | Concurrent batches awaiting acknowledgement (exactly once)      | No       | `2`          | Replicator |
| `coordination.commit.pendingAckQueueSize`  | Max commit batches in the acknowledgement queue (at-least-once) | No       | `5`          | Replicator |

Peak memory from the acknowledgement path is roughly `pendingAckQueueSize × batchSize` records in at-least-once mode, or `parallelizationLimit × batchSize` in exactly once mode. Lower `batchSize` first when memory grows with replication lag. See [Commit tuning](/latest/k2k/configuration-reference/commit-tuning.md) for a worked example.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.lenses.io/latest/k2k/configuration-reference/replication-performance-tuning.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.