I need to create a spark streaming application which should monitor a hdfs path continuously and read only the new incoming csv data in csv tabular format.

[#1]

@Sudharsan Ganeshkumar

Out of the box spark provides the fileStream. You can read more here:

https://spark.apache.org/docs/latest/streaming-programming-guide.html

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#2]

@Sudharsan Ganeshkumar if my answer has helped you please remember to login and mark it as accepted.

[#3 · Accepted solution]

Thanks @Felix Albani. I found it. For Scala code pls check this git hub link https://github.com/phatak-dev/spark2.0-examples/blob/master/src/main/scala/com/madhukaraphatak/examp.... This is exactly what i needed.