Support Questions
Find answers, ask questions, and share your expertise

Spark Streaming inconsistency in reading updated files - using textFileStream

Spark Streaming inconsistency in reading updated files - using textFileStream

Using textFileStream in spark streaming , we are getting the new files from a directory but if any existing file loaded again with content modification in the directory . facing inconsistency like some times those are able to read in spark stream program some times not . what will be the reason and the solution for this .

Thanks in Advance .

1 REPLY 1

Re: Spark Streaming inconsistency in reading updated files - using textFileStream

Rising Star

Spark streaming is reading only new files. You should not change existing files, otherwise Spark Streaming is expected not to work properly.