Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Spark - Filtering Unprocessed File Data

Apache Spark - Filtering Unprocessed File Data



I have a file which keeps getting appended with new lines. Its kind of log file with data in it. I need to read the data from the file store it in database.

My question is, how will i read only unprocessed records from the file and do not read already processed records using Apache Spark framework.


Don't have an account?
Coming from Hortonworks? Activate your account here