Kafka Shrinking/Expanding ISR for partition [TOPIC]

[#1]

I'm seeing a lot in kafka logs with Shrinking and Expanding ISR for partition [TOPIC].

Causing some of the brokers (and partitions) unavailable to producer and consumer clients.

I'm now looking at these configs

replica.fetch.wait.max.ms=500

replica.lag.time.max.ms=10000

replica.lag.max.messages=4000

Should I set these to higher value? What is the best value for these param?

Help is really appreciated. Thanks!

[#2]

@Michael Dennis Uanang

You have to see this

https://kafka.apache.org/082/ops.html

ISR shrink rate

kafka.server:type=ReplicaManager,name=IsrShrinksPerSec

If a broker goes down, ISR for some of the partitions will shrink. When that broker is up again, ISR will be expanded once the replicas are fully caught up. Other than that, the expected value for both ISR shrink rate and expansion rate is 0.

[#3]

Thanks @Neeraj Sabharwal,

Will the config I posted will help? I'm still looking for other reasons why my brokers are going down.

[#4]

@schintalapani

[#5]

@Michael Dennis Uanang Any update on this ? How many kafka brokers you have in your configuration ?

[#6]

Hi @Shishir Saxena

I Have 3 brokers.

Another info:

When 2 brokers are down (logs with ISR Shrinking), the remaining broker will have OOM error.

I adjusted the Heap to 2GB.