__consumer_offsets partition size is huge

[#1]

Hi All,

I have come across a issue where my disk is filling up because of topic __consumer_offsets as i see the size as below.
24G __consumer_offsets-44
29G __consumer_offsets-33
42G __consumer_offsets-13

How do i reduce the size of these partitions, from the logs below is the exception what i am seeing.
2022-04-05 03:03:30,041 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0]: Error due to
java.lang.IllegalArgumentException: inconsistent range

CDH version is 5.16 and kafka is KAFKA-3.1.1-1.3.1.1.p0.2

Assuming that log cleaner can be crashed i tried referring below link but that did not help.
https://my.cloudera.com/knowledge/Disk-utilisation-growing-because-Kafka-LogCleaner-stopped?id=86234

Need your help and advise to fix the issue.

Thanking in advance

[#2]

@shrikantbm Since we will need more details to resolve this, we recommend to raise a support case and get assistance. Thanks.

[#3]

@shrikantbm ,

How old is the oldest log segment on those partitions?

What's the default log retention set for your cluster? Have you set any retention for the __consumer_offsets topic specifically?

Cheers,

André

[#4]

@shrikantbm& team,

Yes, in this case, we need to check cleanup.policy of the topic __consumer_offsets. If the existing cleanup.policy=compact then the log segment of this topic will not be deleted.

You should follow the below steps to conclude and resolve this issue initially. 

1) Check what is current cleanup.policy of the topic __consumer_offsets. You can check it using the command: 

kafka-topics.sh --bootstrap-server <broker-hostname:9092> --describe

OR

kafka-topics.sh --zookeeper <zookeeper-hostname:2181> --describe --topics-with-overrides

Note: topic_name is the name for which you are facing an issue

2)  If you want to clear the old log segment of this topic, then you should set cleanup.policy like cleanup.policy=compact,delete,retention.ms=<30days> 

compact = when the kafka-log is rolled over, it will be compacted

delete - once the offset.retention.ms is reached, the older logs will be removed

retention.ms=<30days>  > the old log segment will be deleted after 30 days.

Note: 30 days are just an example here and this setting will be in ms. You should set it as per your requirement after checking it with the application team and their need. 

For "delete", the property "log.cleaner.enable" must be set to "true"

After configuring this cleanup policy data will be deleted as per retention.ms as suggested above. If you will not set retention.ms then old log segment will be deleted as per retention period set in the CM / Ambari >> kafka >> Conf. The setting is log.retention.hours = <7 Days default> in CM >> Kafka, check what it is in your case so that log segment older than 7 days will be deleted. Kafka will keep checking the old log segment with the help of the property log.retention.check.interval.ms .

Important note: The "delete" on consumer offsets is that you may lose offsets which can lead to duplication/data loss. So check it with your application team before setting a deletion policy.

3) If you still face the same issue, then broker logs need to be reviewed for the root cause of the issue and make the changes accordingly. 

If you found this information helped with your query, please take a moment to log in and click on

KUDOS 🙂 and "Accept as Solution" below this post.

Thank you.