Support Questions

mike_bronson7 · ‎11-30-2017

on all our kafka machines ( production machines ) , we see that: ( no free space )

df -h /var/kafka 

Filesystem      Size  Used Avail Use% Mounted on

/dev/sdb         11T   11T  2.3M 100% /var/kafka

and under /var/kafka/kafka-logs

we see all topic folders (huge size) as example:

117G hgpo.llo.prmt.processed-28 
117G hgpo.llo.prmt.processed-29 
117G hgpo.llo.prmt.processed-3 
117G hgpo.llo.prmt.processed-30 
117G hgpo.llo.prmt.processed-31 
117G hgpo.llo.prmt.processed-32

what is the best approach to delete the topic/s from the folder /var/kafka/kafka-logs ,

and what are the exactly steps to do so , as stop service before deletion etc .

second important question:

what is the mechanizem that suppose to delete automatically the topics ?

Michael-Bronson

dheer_vijji_rag · ‎12-01-2017

@Michael Bronson

1) To find the consumer-group related to a topic you can use the below script:

for i in `/usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh hdpmaster:2181 ls /consumers 2>&1 | grep consumer | cut -d "[" -f2 | cut -d "]" -f1 | cut -d "," -f1` 
 do 
 /usr/hdp/current/kafka-broker/bin/zookeeper-shell.sh hdpmaster:2181 ls /consumers/$i/offsets 2>&1 | grep test  
if [ $? == 0 ]  
then
 echo $i
 fi  
done

2) Yes, setting the retention.ms using ./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic --config retention.ms=1000

3) as @Jordan Moore mentioned there is no automation to delete the topic which is in built to Kafka (we have to follow the process defined above)

4) when I delete a topic we get - Topic <....> is marked for deletion. , is it mean that it will take time until topic will be deleted ?

Yes, it does take time to delete a topic and you can also this message if the delete.topic.enable is set to false (it is the default setting) as given previously:Note: to delete a kafka topic we need to set delete.topic.enable=true it requires kafka service to be restarted

View solution in original post

dheer_vijji_rag · ‎12-02-2017

@Michael Bronson

As you have deleted the topic it looks like the consumer group is deleted, and you can remove the corresponding topic folders (from all nodes) and also the topic from zookeeper using the ZK CLI.

mike_bronson7 · ‎12-02-2017

just to be sure , so it will be safe to remove the topics use rm-rf as ( rm -rf hgpo.llo.prmt.processed-28 . rm -rf hgpo.llo.prmt.processed-28 and so on on all nodes ) ?

Michael-Bronson

mike_bronson7 · ‎12-02-2017

another thing the topics are exist under /brokers/topics/* and not under /consumers/..../.. so in this case I gues we need to remove the topic as rmr /brokers/topics/hgpo.llo.prmt.processed , can you confirm this please

Michael-Bronson

Cloudera Community

Support Questions

what is the safe and best way to delete the kafka topic folders