kafka + how it can be that begin offset is much higher then previous last offset

[#1]

We are using kafka cluster with 3 kafka nodes with kafka version 0.10  ( HDP 2.6.4 )

From the kafka logs we get a very strange message as the following

Beginning offset 1432 is after the ending offset 1298 for topic data.out.cars

note - we have 110 partition for this topic

How it can be that begin offset is 1432 while ending is much less as 1298

What could be wrong in kafka configuration ?

Can we avoid this problem by tune some kafka parameters?

[#2]

@mike_bronson7 

This could be related to a number of reasons, one of them is when moving Kafka data manually (full disk issues for example or replacing disks). In addition to that, there is a property called unclean.leader.election.enable that in kafka HDP 2.6.4 is false by default:

unclean.leader.election.enable

Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss

If you change this property to true and then to false without all brokers in sync, it also may cause the issue you have mentioned. If you want durability over availability, then make sure this property is set to false.