question Re: How to clean up / purge Kafka queues? in Archives of Support Questions (Read Only)

How to clean up / purge Kafka queues?

bsaini — Wed, 21 Oct 2015 21:01:20 GMT

What is the best way to clean / purge the messages in a Kafka queue (irrespective of whether they have been read or not)?

Scenario is that in a dev environment a dummy publisher is pushing messages to a Kafka queue from which a storm topology is reading the messages. If a change has to be made in the publisher such that the current messages in the queue (read or not) are all invalid, so we need to purge those messages and start again.

Re: How to clean up / purge Kafka queues?

Jagatheeshr — Wed, 21 Oct 2015 21:02:47 GMT

For the .8 we are using following

Shutdown Kafka Broker

rm -rf /data/kafka-logs/<topic/Partition_name>

and restart.

Re: How to clean up / purge Kafka queues?

andrewg — Wed, 21 Oct 2015 21:14:06 GMT

In HDP 2.3 you can delete the topic via a standard admin command:

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --delete --topic someTopic

For this to work, one has to make sure the delete.topic.enable flag is set to true (check it via Ambari).

Once deleted, any new message posted to the topic will re-create it, assuming the auto.create.topics.enable property is set to true (it might be disabled in production, same as delete one above). If that's the case, then the admin must create the topic again:

bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --create --topic someTopic --partitions 10 --replication-factor 3

Re: How to clean up / purge Kafka queues?

andrewg — Wed, 21 Oct 2015 21:15:32 GMT

Wouldn't that cause all kinds of issues with zookeeper state? Besides, the data replicas will be on multiple nodes.

Re: How to clean up / purge Kafka queues?

aervits — Wed, 21 Oct 2015 22:36:00 GMT

the topic will still exist until you push new messages to this topic and then follow @AndrewGrande answer below

Re: How to clean up / purge Kafka queues?

hfaouaz — Wed, 21 Oct 2015 23:43:56 GMT

I had better luck expunging the message by changing the retention time and then resetting back.

kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000

Re: How to clean up / purge Kafka queues?

andrewg — Thu, 22 Oct 2015 01:25:22 GMT

Changing a retention period on the topic only marks data segments for deletion. The actual thread which performs an action kicks in every 5 minutes (the default for log.retention.check.interval.ms). Not sure the OP wants to wait up to 5 minutes to delete data.

Re: How to clean up / purge Kafka queues?

bhanu_k_chidiga — Fri, 26 May 2017 15:24:24 GMT

Great info, I am really struggling to understand what happens while consumer reading the messages and rentention time expires for log segment?

Re: How to clean up / purge Kafka queues?

steven_a_royste — Tue, 13 Mar 2018 23:31:39 GMT

@Andrew Grande - When I try to set the log.retention.check.interval configuration setting, I receive an error message stating the following:

Error while executing topic command : Unknown topic config name: log.retention.check.interval.ms

Has this configuration setting been deprecated?

This is the command that I run:

kafka-topics --zookeeper localhost:2181 --alter --entity-type topics --entity-name myTopic --add-config log.retention.check.interval.ms=5000