Application Migration
Understand how to migration applications (consumer groups) from one Kafka cluster to another
Concepts
How it works
Creating an App Migration
Last updated
Was this helpful?
Understand how to migration applications (consumer groups) from one Kafka cluster to another
Replicating data from one Kafka cluster to another is great, but what about the consumers? The offsets don't match so if I blindly re-point my consumer to the new cluster it might reprocess data. This might be ok for your use case, it might not.
If its not, app migrations allow you, in conjunction with K2K replicators, to map the consumer groups from one cluster to another.
Each message has an offset, in the new cluster when replicated it gets an new offset. The source message also has timestamps.
When replicating the K2K apps can add the source offset and timestamp information into the new message headers.
Lenses can then look at the source consumer groups and check when and if its last commited offset is present on the target by looking at the headers. If so creates a new consumer group on the target and sets accordingly the new offset position.
Select the Apps option from the left sidebar, either open the listing for App Migration and click Create App Migration, or click create from the node in the tree panel.
Requires a K2K app with Record timestamp and Tracing Headers offset enabled
Select the Apps option from the left sidebar, either open the listing for App Migration and click Create Application Migration, or click + from the node in the tree panel. Then
Lenses will then check for the corresponding records in the target cluster for the consumers last commited offsets on the source cluster.
Once found, it will create the corresponding consumer group on the target cluster. You can then update your application with the new bootstrap brokers to consume where it left off on the target cluster.
Last updated
Was this helpful?
Was this helpful?

