Bidirectional Replication

Enables bidirectional data replication

K2K can be configured for bidirectional replication, enabling data to flow between two Kafka clusters in both directions while preventing infinite replication loops. When enabled, K2K uses header-based filtering to identify and exclude records that have already been replicated from the opposite direction.

Deployment

Bidirectional replication requires two separate K2K instances to be deployed:

1. Instance 1: Replicates from Cluster A → Cluster B

2. Instance 2: Replicates from Cluster B → Cluster A

Both instances must:

• Use the same pipeline name (if pipeline.mode is set to pipeline)

• Use the same bidirectional configuration (same mode and headers.bidirectional value)

Example

The following example shows the required properties set across the features and bidirectional configuration blocks to enable bidirectional replication.

This configuration must be used by both K2K instances:

# ---------------------------------------------------
# K2K Configuration for Bidirectional Replication
# ---------------------------------------------------

# 1. Global feature flag to enable bidirectional replication.
features:
  bidirectionalReplication: enabled

# 2. Bidirectional configuration block (optional).
bidirectional:
  # Mode determines the identifier used in the bidirectional header.
  # "pipeline" uses the pipeline name; "cluster" uses the source cluster ID.
  mode: cluster
  
  headers:
    # Name of the header used to mark bidirectional records.
    # This header is added to replicated records and checked when filtering.
    bidirectional: "__k2k_bidirectional"

# 3. Pipeline name.
# Must be the same for both instances when bidirectional.mode is set to pipeline
name: bidirectional-pipeline

Bidirectional Modes

The bidirectional mode The choice between pipeline and cluster mode is only relevant when multiple pipelines target the same Kafka cluster pairs.

The mode determines the scope of filtering:

• in pipeline mode, records are only filtered if they were replicated by the same pipeline (identified by pipeline name), allowing different pipelines between the same clusters to operate independently.

• in cluster mode, records are filtered if they originated from the same cluster (identified by cluster ID), regardless of which pipeline replicated them.