Kafka jobs failing : java.lang.AssertionError

[#1]

Our jobs are failing in kafka with error "java.lang.AssertionError: assertion failed: Beginning offset 511 is after the ending offset 510 for topic ps-control partition 6. You either provided an invalid fromOffset, or the Kafka topic has been damaged"

[#2]

@Geoffrey Shelton Okot

please help here.

[#3]

I believe this issue was already resolved. In this particular case, the ISR were not updated appropriately and the offsets on couple of brokers were not in sync with rest of them. Once this issue was fixed, the error was no longer visible.

The offsets not being in sync is attributed to network incident where we had continuous flaps for a long duration of time .

[#4]

How it is fixed?

[#5]

Did you find a solution ?

We had the same issue after a kafka cluster reboot. The spark streaming could not start. It could not read in kafka, with the same error. Our environment : HDP 2.6.2/Kafka 0.10.1/Spark Streaming 2.1 Kafka direct with commitAsync