I am running my structured streaming code in batches with one hour interval and after few batches(complete successfully) offset changed to old offset value and start reading old message again. In log I am getting below warning.
WARN KafkaSource: Partition topicName-29's offset was changed from 1092271004 to 35623, some data may have been missed. Some data may have been lost because they are not available in Kafka any more; either the data was aged out by Kafka or the topic may have been deleted before all the data in the topic was processed. If you want your streaming query to fail on such cases, set the source option "failOnDataLoss" to "true".
I am guessing whenever any incremental data available in kafka topic it is working fine But if there is no incremental data then offset change to older one and start fetching same data which already processed.
Topic is compact topic with retention period of 7 days.
I have tried with bothstartingOffsetsas earliest and latest. Also tried to changefailOnDataLossbut It is not working as expected.