What's new?

This page details the release notes of Lenses K2K replicator.

Version 0.3.0

Summary

  • Centralized Schema registry and Kafka connectivity configuration

    • Schema registry is now specified in source|target.registry.config|headers|cacheSize

    • Kafka configuration is now specified in source|target.kafka.[common|consumer|admin]

  • Kafka connectivity validation (on by default, can be toggled off using features.checkKafkaConnectionOnStartup)

  • com.google.cloud.google-cloud-managedkafka is now packaged with K2K.

  • Added fine grain control for schema replication.

    • configured through schemaMapping.topics

  • Metrics messages and control message are now all part of the same ControlMessage definition hierarchy.

  • Added support for injection of environment variables and/or file contents in the pipeline definition file.

    • the pipeline definition file supports substituting expressions (e.g: ${env:base64:ENV_VARIABLE} by their values.

  • Performance improvements.

Migration

v0.2.0
v0.3.0
Change

.parameters

.configuration

renamed

.coordination.kafka.consumer

.target.kafka.consumer

moved

.coordination.kafka.charset

removed

.coordination.kafka.commit.group

removed

source consumer's group.id is now used .

.coordination.kafka.assignement

.coordination.assignement

removed kafka path segment

.coordination.kafka.commit

.coordination.commit

removed kafka path segment

.source.kafka.connection.servers

.source.kafka.common."bootstrap.servers"

moved

.source.kafka.consumer

inherits all properties defined under source.kafka.common.

.source.kafka.registry.supportedTypes

.schemaMapping.supportedTypes

source and target SR use the same values

.source.kafka.registry.url

.source.registry.config."schema.registry.url"

.source.kafka.registry

.source.registry

.target.kafka.connection.servers

.target.kafka.common."bootstrap.servers"

.target.kafka.consumer

.target.kafka.consumer

inherits all properties defined under target.kafka.common

.target.kafka.producer

.target.kafka.producer

inherits all properties defined under target.kafka.common

.target.kafka.registry.supportedTypes

.schemaMapping.supportedTypes

source and target SR use the same values

.target.kafka.registry.url

.target.registry.config."schema.registry.url"

.target.kafka.registry

.target.registry

.features.offsetCommitOptimizePartition

.features.optimizeOffsetCommitPartition

Example:

name: "simple_pipeline"
source:
  kafka:
    common:
      "bootstrap.servers": "localhost:9092"
      "sasl.mechanism": "PLAIN"
      "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=userName password=pwd;"
      "ssl.truststore.type": "JKS"
      "ssl.truststore.location": "/etc/some/path"
      "ssl.truststore.password": "some-password"
      "security.protocol": "SASL_SSL"
    consumer:
      "group.id": "my-group"
  registry:
    config:
      "schema.registry.url": "${env:raw:REGISTRY_URL}"
      "schema.registry.ssl.truststore.type": "JSK"
      "schema.registry.ssl.truststore.location": "/some/other/path"
      "schema.registry.ssl.truststore.password": "some-sr-password"
target:
  kafka:
    common:
      "bootstrap.servers": "localhost:9099"
      "sasl.mechanism": "PLAIN"
      "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=userName password=pwd;"
      "ssl.truststore.type": "JKS"
      "ssl.truststore.location": "/etc/some/path"
      "ssl.truststore.password": "some-password"
      "security.protocol": "SASL_SSL"
  registry:
    config:
      "schema.registry.url": "localhost:8081"
      "schema.registry.ssl.truststore.type": "JSK"
      "schema.registry.ssl.truststore.location": "/some/other/path"
      "schema.registry.ssl.truststore.password": "some-sr-password"
replication:
  - source:
      name: source
      topic: topic.*
  - sink:
      topic: source
      name: sink-source-topic
      partition: source

Version 0.1.0

Warning

Please note that from this version, a free license token is required. Ensure you obtain one to continue using the service without interruptions. To obtain a free license, send an email to [email protected]. You will receive an automatic reply with the license token information.

Features

Retain Original Message Timestamp

The replication process now includes a feature called keepRecordCreationTimestamp, which retains the original message timestamp. This feature is enabled by default.

Fixes

  • Ensure topic creation waits for Kafka to finish the action before proceeding.

  • Auto-created control topics will use "compact" as the cleanup.policy to help lower costs and decrease startup latency.

Last updated

Was this helpful?