Configuration

Learn how to configure Kafka to Kafka.

Found an issue? Feed it back to us at Github, on Slack, Ask Marios or email.

To execute K2K, you must agree to the EULA and secure a free license.

Accept the EULA by setting license.acceptEula to true .

Secure a free license by:

• emailing [email protected] to receive a token within seconds

• setting license.token with the received token

The K2K application defines its replication behavior using a YAML configuration. This configuration can be structured as a single YAML file or be split across multiple files.

Regardless of a single file or a multi-file structure, the complete, aggregated configuration is validated against a JSON Schema. This validation occurs both at startup and during any runtime updates, ensuring the operational reliability and integrity of your data replication pipelines.

Configuration Structure

The configuration file consists of the following sections:

Required Sections

source - Source cluster configuration Defines connection parameters and Kafka consumer properties for the source cluster. See: Connections

target - Target cluster configuration Specifies connection parameters and Kafka producer properties for the target cluster. See: Connections

replication - Replication flow definitions Contains replication flows that map source topics to target topics and define record routing. See: Selecting topicsTarget PartitionRecords routingSource topics

coordination - State management configuration Configures internal coordination mechanisms, including offset tracking on the target cluster. See: Connections

Optional Sections

features - Feature flags Controls advanced capabilities such as exactly-once semantics, schema replication, and tracing headers. See: Exactly once, Schema Replication

errorHandling - Fault tolerance configuration Defines strategies for handling recoverable errors during replication operations. See: Error Handling

tracing - Record enrichment configuration Configures tracing header injection for observability and data lineage tracking. See: Tracing Headers

Additional Configuration Topics

• Variables and Secrets - Environment variable substitution and credential management. See Variables & Secrets

• Metrics - Monitoring and observability configuration. See Metrics for details.

Schema Definition

The full JSON Schema definition for the configuration is available on GitHub. Refer to it for details on structure, types, and parameter constraints.

Minimal Configuration

name: "mini-k2k"
source:
  kafka:
    connection:
      servers: "sourceKafka:9092"
target:
  kafka:
    connection:
      servers: "targetKafka:9092"
    consumer:
      "group.id": "k2k.mini-k2k.source"
replication:
  - source:
      name: topic-source
      # choose to replicate the topic named my-topic
      topic: "my-topic"
  - sink:
      name: topic-sink
      # use the original topic name as the target topic
      topic: source
      # replicate each record to the same partition as the
      # original topic 
      partition: source
coordination:
  kafka:
    commit:
      # set the consumer group name for reading 
      # the K2K internal coordination topic.
      group: "k2k.mini-k2k.coordination"