Witnessing some unexpected behavior i'm hoping someone might be able to clarify for me.
I have a spark job which we set the zookeeper root to "/example" and the consumer group to "example_group", the offset here is
Using the same consumer group i switched the zookeeper root to "/example_temp" the idea here being that we can continue testing our job on "/example_temp" while the messages are still sent to the kafka topic and when we switch back to "/example" we'll ingest all the messages since the last offset for "/example", allowing us to test a larger load. Now the weird bit is that the offset in "/example_temp" is lower than that of "/example"
And so when I switched back to "/example", i got this error
java.lang.IllegalArgumentException: requirement failed: numRecords must not be negative
Because the offset saved on the broker is that of "/example_temp" and this is lower than the value saved in zookeeper for "/example"
if you're still with me, my question is how did the offset for "/example_temp" end up lower than that of "/example"?