Uses Cases

Use cases for Lenses Kafka to kafka.

We see 4 main use-cases why people replicate Kafka topics across Kafka clusters:

Disaster Recovery (DR)

i.e. to have a plan B.

Use K2K to have a backup Kafka cluster in sync.

If your production goes down (active-passive or active-active), keep a live secondary Kafka cluster. Use K2K to continuously replicate your production cluster data, schemas, and consumer offsets to your secondary DR cluster.

If the main cluster goes down, the consumer applications can simply switch to the DR cluster and resume where they've left off, since the data will have been replicated by K2K.

Migrate

Use K2K to migrate your data and applications to another Kafka.

Your existing Kafka is on-prem, and you want to migrate it to the cloud. Or, maybe you want to switch from one cloud provider to another or a managed Kafka.

Either way, it's not great to stop the world, move the data and applications to the other Kafka and then resume. That would cause business interruption. Instead, it'd be great to have a live gradual migration where the data is replicated continuously to the new Kafka, and applications are gradually moved to the other side.

You can use K2K to do a live migration of Kafka clusters, including schemas and consumer offsets. Once the data and applications have been moved over, the migration is complete, and you can stop the replication. Hence why we call it "one-off".

Extract

i.e. selectively test, simulate & experiment.

Use K2K to safely extract production data to lower environments.

Trying things in production can be risky. It'd be great to run realistic tests, simulations and experiments in a safer environment without affecting production.

The challenge is that production data needs to be filtered and obfuscated before it can be put in a lower environment (e.g. mask PII data).

You can use K2K to continuously and selectively replicate production data to a lower-tier environment such as staging. With K2K, you can filter out entire messages, fields, as well as selectively mask parts of values. You can ensure that the data that lands in the lower environment is safe.

Then you can run multiple versions of your experiments and applications, allowing you to quickly simulate scenarios and figure out systemic streaming bugs.

Share

Use K2K to continuously and safely share data streams across departments and 3rd parties.

Some organisations grow their streaming use to the point that they have separate Kafka clusters per domain or department. To integrate their work, they need to share data live between those Kafka clusters. You can use K2K to continuously integrate events across Kafka topics in your organisation, including their schemas.

In the same spirit, organisations also integrate with 3rd parties in real-time via streaming events. For example, an external partner wishes to tap into purchase events to plan the logistics. They have their own Kafka and which to replicate the events there so that they don't have to integrate all their applications into the organisation's internal Kafka. You can use K2K to share topics with external Kafka clusters while at the same time filtering and obfuscating any sensitive information that should not be public.