For the complete documentation index, see llms.txt. This page is also available as Markdown.

Glossary

Glossary of terms used across K2K documentation.

This page defines the terms used across K2K documentation.

For high-level architecture and how K2K components fit together, see Architecture. For task-oriented configuration guides, see Configuration reference. For which application to deploy, see Deployment.

Naming conventions

Name
What it refers to

K2K

The product—data replication between Kafka clusters

k2k

The replicator application binary (k2k start)

k2k-offset-mapping

The offset mapper application binary (k2k-offset-mapping start)

k2k-tool

The administration CLI and web UI for offset management

Pipeline

The YAML configuration file that defines a replication deployment

K2K

Term
Definition

K2K (Kafka to Kafka)

The suite of applications that replicate data between Kafka clusters. K2K covers data replication, schema migration, and consumer offset translation as a single operational surface.

K2K replicator

The application that reads records from a source Kafka cluster and writes them to a target Kafka cluster. The replicator also handles schema migration when enabled in the pipeline.

K2K offset mapper

The application that reads offset mappings produced during replication and translates consumer group offsets from the source Kafka cluster to the target Kafka cluster. Must run alongside the replicator whenever offset mapping is enabled.

Clusters

Term
Definition

Kafka cluster

A deployment of Apache Kafka brokers that store and serve topic data. Applications connect to a cluster through bootstrap servers.

Source cluster

The Kafka cluster K2K reads from. In most setups this is the cluster holding the data to be copied (for example, production).

Target cluster

The Kafka cluster K2K writes to. In most setups this is the cluster receiving the replicated data (for example a standby, migration destination, or downstream environment).

Kafka clients

Term
Definition

Kafka consumer

A client that reads records from one or more topic partitions and tracks its position through committed offsets. K2K uses consumers on the source Kafka cluster to read data for replication.

Kafka producer

A client that writes records to topic partitions. K2K uses producers on the target Kafka cluster to publish replicated data.

Kafka admin client

A client that manages cluster metadata and administrative operations (for example listing topics, describing consumer groups, or committing offsets). K2K uses admin clients for topic creation, offset mapping, and coordination between instances.

Consumer groups

Term
Definition

Consumer group

A named set of Kafka consumers that cooperate to read from a topic. Kafka assigns each partition to one consumer in the group. Applications track their progress through offsets committed on behalf of the group. K2K's offset mapper watches consumer groups on the source Kafka cluster and translates their committed offsets to the target Kafka cluster.

K2K group

The consumer group the K2K replicator belongs to on the source Kafka cluster. It is configured through source.kafka.consumer."group.id". All replicator instances running the same pipeline share the K2K group. Kafka uses it to distribute source topic partitions across instances. K2K does not commit offsets on the source Kafka cluster; replication progress is tracked through control topics on the target Kafka cluster.

Offsets

Term
Definition

Source offset

The position of a record within a topic partition on the source Kafka cluster. Consumer groups on the source Kafka cluster track their progress using source offsets.

Target offset

The position of a record within a topic partition on the target Kafka cluster. The same record has a different offset on the target Kafka cluster than on the source Kafka cluster, because each Kafka cluster maintains its own offset sequence.

Offset mapping

The correspondence between a source offset and a target offset for a given topic partition. K2K records offset mappings during replication and uses them to translate consumer group positions from the source Kafka cluster to the target Kafka cluster.

Coordination

K2K tracks replication progress and instance ownership through control topics on the target Kafka cluster, rather than committing offsets on the source Kafka cluster.

Term
Definition

Control topics (internal topics)

Topics on the target Kafka cluster used by K2K for internal coordination. They are not application data. Control topics include those used for exclusive assignment, commit, offset mapping, and metrics.

Control consumer

The Kafka consumer on the target Kafka cluster that reads control topics. It keeps each replicator instance aware of partition ownership and replication progress across the K2K group.

Assignment

The process by which Kafka's consumer group protocol distributes source topic partitions across replicator instances in the K2K group. Each instance consumes from the partitions assigned to it.

Exclusive assignment

A K2K mechanism that ensures only one replicator instance actively processes a given source topic partition at a time. During consumer group rebalances, overlapping assignments can occur temporarily. K2K uses a control topic on the target Kafka cluster to grant and enforce exclusive ownership before replication proceeds.

Commit

The act of recording how far replication has progressed on each source topic partition. K2K writes commit records to a control topic on the target Kafka cluster, tracking the last replicated offset per partition.

Delivery semantics

Term
Definition

Exactly once semantics

A delivery guarantee where each record is processed and written once, with no duplicates. In replication, this means each record on the source Kafka cluster appears exactly once on the target Kafka cluster, in the same order.

At least once semantics

A delivery guarantee where each record is delivered one or more times. Duplicates are possible if a failure occurs between writing and committing progress. This is the default when exactly once is not enabled.

Replication

Term
Definition

Replication pipeline

The unit of configuration in K2K. A pipeline names a source Kafka cluster, a target Kafka cluster, the features to enable, and the replication stages that define what to copy and where records land. Each K2K application runs one pipeline definition.

Replication stage

A step in the pipeline's replication definition. The first stage is always a source stage, which declares which topics to read from the source Kafka cluster. The following stages are sink stages, which declare how records are routed to the target Kafka cluster (topic name, partition assignment). A pipeline requires at least one source stage and one sink stage.

Last updated

Was this helpful?