Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Streaming inconsistency in reading updated files - using textFileStream

Highlighted

Spark Streaming inconsistency in reading updated files - using textFileStream

Using textFileStream in spark streaming , we are getting the new files from a directory but if any existing file loaded again with content modification in the directory . facing inconsistency like some times those are able to read in spark stream program some times not . what will be the reason and the solution for this .

Thanks in Advance .

1 REPLY 1

Re: Spark Streaming inconsistency in reading updated files - using textFileStream

Rising Star

Spark streaming is reading only new files. You should not change existing files, otherwise Spark Streaming is expected not to work properly.

Don't have an account?
Coming from Hortonworks? Activate your account here