Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HiveContext trying to read dropped data

HiveContext trying to read dropped data

New Contributor

I have a spark application which drops a partition, recreates it (dynamically) and populates it with new data.

When i run it the first time (partition does not exist), it works fine. The next time i run it (partition is dropped) the application tries to read data from old partition, from one random file, but fails because the file does not exist.

This problem occurs sporadically with no observed pattern. Any idea why it is happening?

Don't have an account?
Coming from Hortonworks? Activate your account here