Support Questions
Find answers, ask questions, and share your expertise

__consumer_offsets partition size is huge

New Contributor

Hi All,

 

I have come across a issue where my disk is filling up because of topic __consumer_offsets as i see the size as below.
24G __consumer_offsets-44
29G __consumer_offsets-33
42G __consumer_offsets-13

How do i reduce the size of these partitions, from the logs below is the exception what i am seeing.
2022-04-05 03:03:30,041 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0]: Error due to
java.lang.IllegalArgumentException: inconsistent range

CDH version is 5.16 and kafka is KAFKA-3.1.1-1.3.1.1.p0.2

Assuming that log cleaner can be crashed i tried referring below link but that did not help.
https://my.cloudera.com/knowledge/Disk-utilisation-growing-because-Kafka-LogCleaner-stopped?id=86234

 

Need your help and advise to fix the issue.

Thanking in advance

2 REPLIES 2

Moderator

@shrikantbm Since we will need more details to resolve this, we recommend to raise a support case and get assistance. Thanks.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

Master Collaborator

@shrikantbm ,

 

How old is the oldest log segment on those partitions?

What's the default log retention set for your cluster? Have you set any retention for the __consumer_offsets topic specifically?

 

Cheers,

André

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.
; ;