Roadmap

A list of planned features

Data obfuscation

When migrating data to a lower tier cluster, it is crucial to safeguard sensitive information. The solution will provide options to redact or exclude such data effectively.

Data Filtering and Message Reshaping

Data filtering and message reshaping enables selective data transformation when copying between Kafka topics. You can apply custom filters to match specific business logic, restructure message formats, and derive new fields from existing data. This feature supports cross-cluster data migration, topic migration within the same cluster, and real-time data processing pipelines where transformation is required alongside data movement.

Dynamic Message Routing

Enable the replicator to inspect the data within a Kafka message and dynamically choose its destination. This powerful routing capability allows you to direct messages across topics, whether they are in the same cluster or a remote one. The primary use case is to intelligently split a single data stream, simplifying producer logic and enabling more flexible data architectures. For example, from a single source topic, you can route EU-specific data to a local topic while simultaneously sending US-specific data to a topic on a different target cluster.

Support for Glue Schema Registry

It will add native support for the AWS Glue Schema Registry. This feature will automate the replication of schemas in lockstep with message data, eliminating the complex and error-prone manual processes typically required when migrating schemas between different environments. The primary goal is to streamline migrations to and from AWS MSK, ensuring data integrity and preventing deserialization errors by keeping schemas perfectly synchronized between the source and target clusters.

Continuous Topic Configuration and Partition Sync

The replicator, if configured, will automatically synchronize topic configurations from the source to the target cluster. Building on the existing auto-create topic feature, the replicator will detect and apply changes to partition counts and other topic settings as they happen on the source.

Consumer Offsets Replication

While the consumer offset topic can be replicated in its raw format, it might not work in scnearios where target topic-partition offset does not match between source and target Kafka cluster. Consider the scenario where replication is started to an empty cluster, at source the topic-partition retention has already deleted records. So a record on parition 2 and offset 1234, will be published as offset 0 in the target cluster. The Consumer Offsets replication will take care about the mapping, and ensure the consumer group has the correct latest offset in the target.