Exactly Once Delivery

Enable exactly once delivery guarantees.

Found an issue? Feed it back to us at Github, on Slack, Ask Marios or email.

To execute K2K, you must agree to the EULA and secure a free license.

Accept the EULA by setting license.acceptEula to true .

K2K uses at-least-once delivery by default, but it can be configured for exactly-once delivery. This tutorial will guide you on how to enable exactly-once delivery.

Requirements

This tutorial assumes the following files exist (See Setting up for more details):

k2k-pipeline.yml

name: "my-first-replication"
features:
  autoCreateControlTopics: enabled
  autoCreateTopics: enabled
source:
  kafka:
    common:
      "bootstrap.servers": "kafka-source:9092"
    consumer:
      "group.id": "k2k.my-first-k2k"
target:
  kafka:
    common:
      "bootstrap.servers": "kafka-target:9092"
replication:
  - source:
      topic: ".*"
  - sink:
      topic: source
      partition: source

docker-compose.yml

To ensure a clean start, execute this command to reset any prior configurations from earlier tutorials.

docker compose down

Enabling Exactly Once

To achieve exactly-once semantics, enable that option and set the idempotent flag on the producer instance as follows:

features:
  exactlyOnce: enabled
  ....
target:
  kafka:
    producer:
      "enable.idempotence": "true"
  ....

k2k-pipeline.yml

name: "my-first-replication"
features:
  autoCreateControlTopics: enabled
  autoCreateTopics: enabled
  exactlyOnce: enabled
source:
  kafka:
    common:
      "bootstrap.servers": "kafka-source:9092"
    consumer:
      "group.id": "k2k.my-first-k2k"
target:
  kafka:
    common:
      "bootstrap.servers": "kafka-target:9092"
    producer:
      "enable.idempotence": "true"
replication:
  - source:
      topic: ".*"
  - sink:
      topic: source
      partition: source

See it in action

docker compose up -d kafka-source kafka-target

#create a topic
docker compose exec kafka-source \
      ./opt/kafka/bin/kafka-topics.sh \
      --create \
      --topic user-topic \
      --partitions 5 \
      --bootstrap-server 127.0.0.1:9092 

# add some data to the source topic
docker-compose exec kafka-source \
    ./opt/kafka/bin/kafka-producer-perf-test.sh \
    --topic user-topic \
    --num-records 100 \
    --record-size 20 \
    --throughput -1 \
    --producer-props bootstrap.servers=localhost:9092

docker compose up -d k2k

#inspect the target topic
docker compose exec kafka-target \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9099 \
  --topic user-topic \
  --property print.key=true \
  --property key.separator=, \
  --property "print.offset=true" \
  --property "print.partition=true" \
  --from-beginning

#compare both source and target topics 

#target instance records
docker compose exec kafka-target \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9099 \
  --topic user-topic \
  --property print.key=true \
  --property key.separator=, \
  --property "print.offset=true" \
  --property "print.partition=true" \
  --from-beginning  

#source instance records
docker compose exec kafka-source \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic user-topic \
  --property print.key=true \
  --property key.separator=, \
  --property "print.offset=true" \
  --property "print.partition=true" \
  --from-beginning