Uses Cases

Use cases for Lenses Kafka replicator.

Here are four key business-driven reasons for replicating Kafka topics across clusters:

Disaster Recovery (DR)

Disaster Recovery (DR) is the strategy for ensuring business continuity by maintaining a standby Kafka cluster that can take over if the primary system fails. A robust DR plan minimizes data loss and service downtime during an outage.

To implement DR, you can maintain a secondary Kafka cluster that is kept in sync with your primary (production) cluster. This is often achieved using a replication tool like K2K (Kafka-to-Kafka).

K2K continuously replicates the essential components from the primary to the secondary DR cluster, including:

• Topic Data: All messages from your Kafka topics.

• Schemas: The schemas stored in your Schema Registry.

• Consumer Offsets: The last committed position for each consumer group. (Note, for now K2K can replicate the raw __consumers_offsets topic, without any offsets mapping when the source and target cluster topic offsets are not aligned)

This setup can support both active-passive (where the DR cluster is on standby) and active-active (where both clusters handle live traffic) configurations.

Seamless Kafka Migration with K2K

K2K enables smooth migration of data and applications between Kafka environments. Whether you're moving an on-premises Kafka to the cloud, switching between cloud providers or managed Kafka services, or migrating from production to staging or development environments, K2K makes the transition effortless without disrupting business operations.

Key Benefits

• Live, Gradual Migration: Avoid downtime by continuously replicating data to the new Kafka environment. Gradually move applications while maintaining operations.

• Comprehensive Support: Migrate Kafka clusters, including schemas and consumer offsets, ensuring a complete and synchronized transfer.

• Flexible Environment Options: Seamlessly migrate between production, staging, and development environments to support testing and deployment needs.

• Easy Completion: Once migration is complete, simply stop the replication, finalizing the process efficiently.

Leverage K2K for a hassle-free, one-time migration.

Data Extraction and Simulation

When testing, simulation, and experimentation are necessary, K2K offers a way to securely extract production data for use in lower environments. Running tests in production can be risky; hence, it is beneficial to conduct realistic tests and simulations in a safer environment, ensuring no disruption to live services. The primary challenge is the requirement to filter and obfuscate production data (e.g., masking PII data) before deploying it to environments like staging.

Key Features of K2K

• Selective Replication: Continuously and selectively replicate only the required production data to lower-tier environments.

• Data Filtering: Filter out complete messages or specific fields that are unnecessary or sensitive.

• Data Masking: Selectively mask portions of data values to maintain data privacy.

• Environmental Safety: Ensure that no sensitive information is compromised while using obfuscated data in non-production settings.

.

Data Stream Sharing for Kafka

K2K provides a reliable solution for sharing data streams across departments and with third parties. As organizations evolve, they often require separate Kafka clusters for different domains or departments. To facilitate seamless integration, K2K enables live data sharing between these Kafka clusters. It ensures continuous event integration across Kafka topics, along with their schemas.

Additionally, organizations can use K2K to collaborate with external partners through real-time streaming events. For instance, an external partner may wish to access purchase events to optimize their logistics. Instead of integrating all their applications into the organization's Kafka system, they can replicate events using K2K into their own Kafka cluster. K2K allows for topic sharing with external clusters, while simultaneously filtering and obfuscating sensitive information to protect privacy.