plugConnections

K2K Offset Mapping App connection details

circle-exclamation

Configuration Parameters

K2K Offset Mapping provides both the flexibility to individually configure each module (Consumers, Producers, Admin Clients) as well as the ability to provide common configuration only once. To achieve this, under source.kafka and target.kafka a common key can be specified which allows settings to be specified for all Consumers, Producers (if applicable) and Admin Clients for those clusters.

The full list of available configuration keys is listed in the table bellow.

source.kafka.common

A set of properties that will be inherited by source.kafka.consumer and source.kafka.admin

It's advisable to define "bootstrap.servers" and any other common properties here to avoid repetition.

source.kafka.admin

Allows any standard Apache Kafka Admin Client property to be provided. It will inherit all properties from source.kafka.common .

If not defined in common, the following property must be provided:

  • "bootstrap.servers"

target.kafka.common

A set of properties that will be inherited by target.kafka.consumer and taget.kafka.admin

It's advisable to define "bootstrap.servers" and any other common properties here to avoid repetition.

target.kafka.producer

Allows any standard Apache Kafka Producer property to be provided. It will inherit all properties provided in target.kafka.common

​If not defined in common, the following property must be provided:

  • "bootstrap.servers"

target.kafka.consumer

Allows any standard Apache Kafka Consumer property to be provided. It will inherit all properties provided in target.kafka.common

​If not defined in common, the following property must be provided:

  • "bootstrap.servers"

target.kafka.admin

Allows any standard Apache Kafka Admin Client property to be provided. It will inherit all properties from target.kafka.common .

​If not defined in common, the following property must be provided:

  • "bootstrap.servers"

Connection Types

Below is a list of connection settings for configuring your application. In the examples, both source and target clusters are configured using the same provider (Confluent, AWS, ...) . K2K Offset Mapping however supports replication pipelines where source and target cluster are different.

Confluent Cloud

target:
  kafka:
    common:
      "bootstrap.servers": "<Confluent Cloud Kafka Bootstrap Brokers>"
      "security.protocol": "SASL_SSL"
      "sasl.jaas.config": "<Confluent Cloud Kafka client jaas configuration>" 
      "sasl.mechanism": "PLAIN"
      "acks": "all"
      
source:
  kafka:
    common:
      "bootstrap.servers": "<Confluent Cloud Kafka Bootstrap Brokers>"
      "security.protocol": "SASL_SSL"
      "sasl.jaas.config": "<Confluent Cloud Kafka client jaas configuration>"
      "sasl.mechanism": "PLAIN"
      "client.dns.lookup": "use_all_dns_ips"

AWS MSK

Google Managed Kafka

When replicating to a Google Managed Kafka with automatic topic creation enabled, ensure this setting is included. This prevents errors due to Google Managed Kafka's restriction on changing default topic settings. The Kafka Admin Client does not clearly indicate the topic setting policy when it responds to topic configuration changes, and it leads to the application failing and stopping.

SSL

Plain Text

Example

The following example demonstrates how these properties are structured in the configuration file. This single block shows the required settings, an optional Schema Registry connection, and custom consumer properties for performance tuning.

Last updated

Was this helpful?