> For the complete documentation index, see [llms.txt](https://docs.lenses.io/latest/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lenses.io/latest/k2k/deployment.md).
# Deployment
Which K2K application to deploy and when.
Replicating between Kafka clusters means consuming from one broker set and producing to another. Kafka treats these as independent systems, so we need separate processes to copy records and, when failover matters, to translate consumer positions.
K2K ships as two applications that share one pipeline file. The replicator copies records from the source Kafka cluster to the target Kafka cluster; the offset mapper translates consumer group positions so applications can resume from equivalent offsets on the target Kafka cluster. See [Architecture](/latest/k2k/architecture.md) for how applications, Kafka clients, and the pipeline file fit together. Every deployment must accept the EULA; a [license](/latest/k2k/configuration-reference/licenses.md) token is needed to go beyond the free tier partition cap or to use a licensed feature. Both applications share the same command-line options on `start`; see [Command-line options](#command-line-options) below. To layer secrets over a base definition, see [Minimal Configuration](/latest/k2k/minimal-configuration.md#loading-pipeline-files).
## The two applications
| Application | Command | Purpose |
| ----------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| K2K replicator | `k2k start` | Reads records from the source Kafka cluster and writes them to the target Kafka cluster. Handles schema migration, topic creation, exactly once delivery, and other replication features configured in the pipeline. |
| K2K offset mapper | `k2k-offset-mapping start` | Watches consumer groups on the source Kafka cluster and translates their committed offsets to equivalent positions on the target Kafka cluster, using offset mappings recorded by the replicator. |
The replicator is required for every pipeline. The offset mapper is only needed when [offset mapping](/latest/k2k/configuration-reference/offset-mapping.md) is enabled — but when it is enabled, **both applications are required**. Enabling `features.offsetMapping` in the pipeline file does not translate consumer offsets by itself.
| Requirement | Deploy |
| ---------------------------- | ------------------------------------------ |
| Replication only | `k2k start` |
| Replication + offset mapping | `k2k start` and `k2k-offset-mapping start` |
## When to deploy each application
{% hint style="warning" %}
When `features.offsetMapping` is `enabled`, deploy and run both `k2k` and `k2k-offset-mapping`. Enabling the feature in the pipeline file does not translate consumer offsets by itself.
{% endhint %}
### Replicator only
Deploy the replicator when you need data replicated between Kafka clusters and do not need consumer group positions translated on the target Kafka cluster.
Typical scenarios:
* Disaster recovery standby clusters
* Test, staging, or analytics feeds from production
* [Bidirectional replication](/latest/k2k/configuration-reference/bidirectional-replication.md) (one replicator per direction)
* Migrations where applications will be redeployed on the target Kafka cluster with fresh consumer groups
```shell
k2k start -f my_pipeline.yml
```
### Offset mapping (both applications required)
Deploy **both** applications when `features.offsetMapping` is enabled and applications on the target Kafka cluster must resume from positions equivalent to their source consumer groups. Keep both processes running continuously.
Typical scenarios:
* Failover where consumers switch to the target Kafka cluster without reprocessing from the beginning
* Gradual migration where applications move to the target Kafka cluster one at a time while both Kafka clusters stay in sync
The replicator records source-Kafka-cluster-to-target-Kafka-cluster offset mappings as it replicates. The offset mapper reads those mappings and commits translated offsets on the target Kafka cluster. There is no requirement to start the replicator before the offset mapper; if the replicator is not running, the offset mapper simply has no mappings to translate yet.
```shell
k2k start -f my_pipeline.yml
k2k-offset-mapping start -f my_pipeline.yml
```
See [Offset mapping](/latest/k2k/configuration-reference/offset-mapping.md) for configuration.
### Starting the offset mapper alone
Starting the offset mapper without the replicator is not a valid offset-mapping deployment. The replicator must be running with `features.offsetMapping: enabled` to record mappings; without it, the offset mapper has nothing to translate. Running the offset mapper alone is only useful for inspection or testing.
## Command-line options
Both applications are started with a subcommand followed by options. Global flags (`--help`, `--version` / `-v`) are available on the top-level command.
### Replicator
```shell
k2k start [options]
k2k dry-run [options]
```
`start` runs the pipeline against Kafka. `dry-run` runs the same pipeline but prints replicated records to stdout and disables offset commits. Use `dry-run` to validate routing and schema migration without writing to the target Kafka cluster. Telemetry is not available on `dry-run`.
### Offset mapper
```shell
k2k-offset-mapping start [options]
```
### Options
These options apply to `k2k start`, `k2k dry-run`, and `k2k-offset-mapping start`. The `--telemetry` / `-t` flag is only available on `start` subcommands (not `dry-run`).
| Option | Short | Default | Description |
| --------------------------- | ----- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--file` | `-f` | *(required)* | Pipeline definition file. Pass multiple times to merge files; later files override earlier ones. See [Loading pipeline files](/latest/k2k/minimal-configuration.md#loading-pipeline-files). |
| `--defaults` | `-d` | Classpath defaults | HOCON file with default pipeline values. Pass multiple times to layer files; later files override earlier ones. |
| `--template-engine` | `-g` | `enabled` | Enable or disable `${...}` template resolution before YAML parsing. Set to `disabled` when pipeline files contain no templates. See [Secrets and templates](/latest/k2k/configuration-reference/secrets-and-templates.md). |
| `--telemetry` | `-t` | disabled | Enable OpenTelemetry metric export. Configure exporters through standard `OTEL_*` environment variables. See [Observability](/latest/k2k/configuration-reference/observability.md). |
| `--termination-log` | `-l` | `/dev/termination-log` | File to write a shutdown or error message on exit. The default path is the Kubernetes termination log location. |
| `--termination-log-enabled` | `-w` | `enabled` | Write the termination log on exit. Set to `disabled` to turn this off. |
| `--verbosity` | | | Log level: `off`, `error`, `info`, `debug`, or `trace`. When omitted, the logging configuration from the environment is used. |
### Examples
Start the replicator with a base pipeline and a secrets overlay, telemetry enabled, and debug logging:
```shell
k2k start -f base_pipeline.yml -f secrets_overrides.yml -t --verbosity debug
```
Start the offset mapper with the same pipeline files:
```shell
k2k-offset-mapping start -f base_pipeline.yml -f secrets_overrides.yml
```
Validate a pipeline without writing to Kafka:
```shell
k2k dry-run -f my_pipeline.yml
```
Run `k2k start --help`, `k2k dry-run --help`, or `k2k-offset-mapping start --help` for the exact syntax accepted by your installed version.
## Scaling
Both applications can run as multiple instances with the same pipeline configuration to increase throughput. Each instance is independent; Kafka's consumer group protocol distributes work for the replicator, while additional offset mapper instances process consumer group translations in parallel.
| Application | Scaling |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Replicator | Run multiple instances with the same pipeline configuration. Kafka distributes source topic partitions across the [K2K group](/latest/k2k/configuration-reference/consumer-group.md). See [Horizontal scaling](/latest/k2k/configuration-reference/horizontal-scaling.md). |
| Offset mapper | Run multiple instances with the same pipeline configuration to translate consumer groups in parallel. See [Horizontal scaling](/latest/k2k/configuration-reference/horizontal-scaling.md). |
## Prerequisites
* [License accepted](/latest/k2k/configuration-reference/licenses.md)
* Pipeline file with source and target Kafka cluster connections
* Both source and sink replication stages in `replication` — see [Topic selection and data routing](/latest/k2k/configuration-reference/topic-selection-and-data-routing.md)
* For offset mapping: replicator and offset mapper both running with `features.offsetMapping: enabled` in the pipeline, `target.kafka.consumer."group.id"` set, and a license token carrying the offset mapping entitlement
## Deployment recommendation
Deploy K2K replicator instances close to the **target** Kafka cluster. Producer writes are latency-sensitive — co-location reduces timeouts, improves throughput, and strengthens exactly-once delivery guarantees.
## Install
See [Releases and compatibility](/latest/k2k/deployment/releases-and-compatibility.md) for supported versions and upgrade guidance.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.lenses.io/latest/k2k/deployment.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
