__consumer_offsets partition size is huge

[#1]

Hi All,

I have come across a issue where my disk is filling up because of topic __consumer_offsets as i see the size as below.
24G __consumer_offsets-44
29G __consumer_offsets-33
42G __consumer_offsets-13

How do i reduce the size of these partitions, from the logs below is the exception what i am seeing.
2022-04-05 03:03:30,041 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0]: Error due to
java.lang.IllegalArgumentException: inconsistent range

CDH version is 5.16 and kafka is KAFKA-3.1.1-1.3.1.1.p0.2

Assuming that log cleaner can be crashed i tried referring below link but that did not help.
https://my.cloudera.com/knowledge/Disk-utilisation-growing-because-Kafka-LogCleaner-stopped?id=86234

Need your help and advise to fix the issue.

Thanking in advance

[#2]

@shrikantbm Since we will need more details to resolve this, we recommend to raise a support case and get assistance. Thanks.

[#3]

@shrikantbm ,

How old is the oldest log segment on those partitions?

What's the default log retention set for your cluster? Have you set any retention for the __consumer_offsets topic specifically?

Cheers,

André