I have a spark application which drops a partition, recreates it (dynamically) and populates it with new data.
When i run it the first time (partition does not exist), it works fine. The next time i run it (partition is dropped) the application tries to read data from old partition, from one random file, but fails because the file does not exist.
This problem occurs sporadically with no observed pattern. Any idea why it is happening?