This page describes a tutorial for deleting data in compacted topics from SQL Studio.
In this example, we will show how we can use Lenses to delete records from a compacted topic which stores users' calling information based on a different topic that stores the users’ response to a “do you want to be contacted back” survey.
With the following code, we can create a compacted topic that holds user call information:
Notice we add cleanup.policy=compact to tell Lenses we want the topic to be compacted. The remaining properties try to force compaction to happen often so that we can easily visualize the result (this should not be used in production though).
We start by adding some records to our user_info topic:
Which we can see by inspecting the topic.
We additionally add a second topic which will hold information regarding users' response to the survey:
As previously, we added some records to contact_survey
Using the following:
We are essentially issuing a delete command for all users who said they didn’t want to be contacted.
Looking at our user_calls topic again, we can see the newly inserted records with a null value, but our original records are still there… How so?
Due to some internals of Kafka, log compaction doesn’t always work immediately but in this case, by adding an extra record we can force it to happen:
Looking at the data inside our topic, we can now see that users who responded that they didn’t want to be contacted are no longer part of our topic; the tombstone records (the ones with a null value) will stay around for as long as our retention policy specifies and will eventually be removed, leaving us with a topic with only users that want to be contacted.