Kafka to Kafka Replicator
This page describes how to manage Kafka to Kafka data replication in Lenses.
The K2K application is designed to facilitate seamless data replication between Kafka clusters while also replicating topic metadata. This capability ensures consistency across different environments or data centers. K2K supports a range of use cases including disaster recovery, data migration, and cross-datacenter data replication. By preserving both data and metadata, K2K enables a robust and efficient solution for maintaining synchronized Kafka clusters. Follow this link to learn about the concepts.
Lenses streamlines K2K application management by simplifying creation and deployment, leading to faster and more efficient workflows. With self-service capabilities through Kafka Connections, users gain autonomy, improving productivity. Observability is enhanced through detailed throughput metrics, providing insights for optimized performance. Comprehensive governance is maintained with IAM policies, ensuring security and compliance.
Creating an application
For the moment, Lenses integration supports a subset of the K2K features. For example, via Lenses the user can replicate data to topics matching the source cluster. K2K support more complex topic name routing as described in the standalone version.
The K2K application, integrated within Lenses, is configurable through a YAML file. The user interface offers auto-complete functionality to streamline YAML navigation, enhancing user efficiency. Alternatively, users can configure the application using the UI components.
To create the application, follow these steps.
Delete the K2K application
Navigate to the application details page and select "Delete" from the actions menu to remove the application. This will delete it from both Lenses and its Kubernetes deployment.
Configuration Reference
This table outlines the complete set of fields for the K2K resource, which is utilized to define and configure a Kafka-to-Kafka application. The configuration is divided into two sections:
status: Reflects the most recently observed state; it is read-only.
spec: Depicts the intended state of the K2K application.
spec.sourceKafkaLRN
Sets the source KafkaConnection LRN to read from.
string
(none)
Yes
spec.targetKafkaLRN
Sets the target KafkaConnection LRN to write to.
string
(none)
Yes
spec.topics
Sets the topic(s) to replicate.
array (of string)
(none)
Yes
spec.deploymentNamespaceLRN
Defines the Kubernetes namespace to deploy to.
string
(none)
Yes
spec.kubernetes
Defines Kubernetes deployment overrides. (Ref: K8sDeploymentOverrides)
object
(none)
No
spec.replicator
Provides additional control knobs for the K2K replicator.
object
(none)
No
spec.paused
Halts replication by scaling down to zero pods.
boolean
false
No
spec.replicator.schemaMapping
Enables schema replication.
boolean
false
No
spec.replicator.exactlyOnce
Enables exactly-once processing.
boolean
false
No
spec.replicator.autoCreateTopics
Enables auto creation of topics.
boolean
false
No
spec.replicator.autoCreateControlTopics
Enables auto creation of control topics.
boolean
false
No
spec.replicator.keepRecordCreationTimestamp
Enables preserving record timestamps.
boolean
false
No
status.updatedAt
Contains the time instant this object was updated.
string (date-time)
(none)
Yes
status.state
The high-level state of the application. (Ref: K2KAppState)
string
(none)
Yes
status.statusLastTransitionAt
Contains the time instant the status was changed.
string (date-time)
(none)
Yes
status.statusMessage
Is a human-readable message indicating details about the last transition.
string
(none)
Yes
status.observedGeneration
Reflects the generation of the spec that this status corresponds to.
integer
(none)
Yes
status.deployment
Provides information about the Kubernetes deployment.
object
(none)
Yes
status.kafka
Provides information about the Kafka-related status.
object
(none)
Yes
status.deployment.inSync
Indicates if the deployed resources match the desired state.
boolean
(none)
Yes
status.deployment.managedResources
A list of resources managed by the deployment. (Ref: KubernetesDeploymentStatusResource)
array
(none)
Yes
status.deployment.pods
The status of the individual pods in the deployment. (Ref: KubernetesDeploymentStatusPod)
array
(none)
Yes
status.kafka.consumerGroup
Holds the name of the consumer group that this K2KApp uses.
string
(none)
Yes
Last updated
Was this helpful?