Application Migration

Understand how to migration applications (consumer groups) from one Kafka cluster to another

Concepts

Replicating data from one Kafka cluster to another is great, but what about the consumers? The offsets don't match so if I blindly re-point my consumer to the new cluster it might reprocess data. This might be ok for your use case, it might not.

If its not, app migrations allow you, in conjunction with K2K replicators, to map the consumer groups from one cluster to another.

How it works

Each message has an offset, in the new cluster when replicated it gets an new offset. The source message also has timestamps.

When replicating the K2K apps can add the source offset and timestamp information into the new message headers.

Lenses can then look at the source consumer groups and check when and if its last commited offset is present on the target by looking at the headers. If so creates a new consumer group on the target and sets accordingly the new offset position.

Creating an App Migration

Select the Apps option from the left sidebar, either open the listing for App Migration and click Create App Migration, or click create from the node in the tree panel.

Requires a K2K app with Record timestamp and Tracing Headers offset enabled

Select the Apps option from the left sidebar, either open the listing for App Migration and click Create Application Migration, or click + from the node in the tree panel. Then

1

Add name

Give the migration a name

2

Select the K2K replicator

Choose the replicator that is replicating the topics your consumer group is reading from

3

Select the consumer group

Select the consumer group to migrate

4

Stop the consumer

The consumer group needs to be inactive to the source offsets are not a moving target

5

Confirm you have stopped the consumer

Once stopped, confirm to begin the mirgation process

Lenses will then check for the corresponding records in the target cluster for the consumers last commited offsets on the source cluster.

Once found, it will create the corresponding consumer group on the target cluster. You can then update your application with the new bootstrap brokers to consume where it left off on the target cluster.