Kafka to Kafka Replicator

Understand how to manage Kafka to Kafka data replication in Lenses.

The K2K application is designed to facilitate seamless data replication between Kafka clusters while also replicating topic metadata. This capability ensures consistency across different environments or data centers. K2K supports a range of use cases including disaster recovery, data migration, and cross-datacenter data replication. By preserving both data and metadata, K2K enables a robust and efficient solution for maintaining synchronized Kafka clusters. To learn more concepts see Overview.

Lenses streamlines K2K application management by simplifying creation and deployment, leading to faster and more efficient workflows. With self-service capabilities through Kafka Connections, users gain autonomy, improving productivity. Observability is enhanced through detailed throughput metrics, providing insights for optimized performance. Comprehensive governance is maintained with IAM policies, ensuring security and compliance.

Creating a K2K application

Select the Apps option from the left sidebar, either open the listing for K2K Replicators and click Create K2K Replicator, or click + from the node in the tree panel. Then

1

Name and metadata

Add a name and metadata

2

Select source, target and deployment environment

Source is where the source target is, target, the environment to write to, and deployment environment is where the application is to be deployed

Deployment environment should be as close to the target as possible, prefereably in the target environment. See Kafka Connections

3

Select topics

Select one or more topics to replicate

4

Optional add topic mapping

Decide how you want to map the topics on the target cluster

5

Optional set replication options

These settings allow you to control what is replicated, for example message headers and timestamps and auto creation settings on the target.

Configuration Reference

This table outlines the complete set of fields for the K2K resource, which is utilized to define and configure a Kafka-to-Kafka application. The configuration is divided into two sections:

• status: Reflects the most recently observed state; it is read-only.

• spec: Depicts the intended state of the K2K application.

Field	Description	Type	Default	Required
spec.sourceKafkaLRN	Sets the source KafkaConnection LRN to read from.	string	(none)	Yes
spec.targetKafkaLRN	Sets the target KafkaConnection LRN to write to.	string	(none)	Yes
spec.topics	Sets the topic(s) to replicate.	array (of string)	(none)	Yes
spec.deploymentNamespaceLRN	Defines the Kubernetes namespace to deploy to.	string	(none)	Yes
spec.kubernetes	Defines Kubernetes deployment overrides. (Ref: K8sDeploymentOverrides)	object	(none)	No
spec.replicator	Provides additional control knobs for the K2K replicator.	object	(none)	No
spec.paused	Halts replication by scaling down to zero pods.	boolean	false	No
spec.replicator.schemaMapping	Enables schema replication.	boolean	false	No
spec.replicator.exactlyOnce	Enables exactly-once processing.	boolean	false	No
spec.replicator.autoCreateTopics	Enables auto creation of topics.	boolean	false	No
spec.replicator.autoCreateControlTopics	Enables auto creation of control topics.	boolean	false	No
spec.replicator.keepRecordCreationTimestamp	Enables preserving record timestamps.	boolean	false	No
status.updatedAt	Contains the time instant this object was updated.	string (date-time)	(none)	Yes
status.state	The high-level state of the application. (Ref: K2KAppState)	string	(none)	Yes
status.statusLastTransitionAt	Contains the time instant the status was changed.	string (date-time)	(none)	Yes
status.statusMessage	Is a human-readable message indicating details about the last transition.	string	(none)	Yes
status.observedGeneration	Reflects the generation of the spec that this status corresponds to.	integer	(none)	Yes
status.deployment	Provides information about the Kubernetes deployment.	object	(none)	Yes
status.kafka	Provides information about the Kafka-related status.	object	(none)	Yes
status.deployment.inSync	Indicates if the deployed resources match the desired state.	boolean	(none)	Yes
status.deployment.managedResources	A list of resources managed by the deployment. (Ref: KubernetesDeploymentStatusResource)	array	(none)	Yes
status.deployment.pods	The status of the individual pods in the deployment. (Ref: KubernetesDeploymentStatusPod)	array	(none)	Yes
status.kafka.consumerGroup	Holds the name of the consumer group that this K2KApp uses.	string	(none)	Yes