I am trying to read twitter data into my hdfs path from kafka using spark streaming application.
Both my kafka and spark streaming apps are running fine, but spark app Is not able to write any data in to destination folder. somehow it is creating a Success file in the destination folder but not the actual data.
I' m using
Kafka 0.10.0.2.5 Spark 1.6.x.2.5 HDP2.5 version
Anyhelp is appreciated. I am Sharing my code.
Check that Spark user HDFS privileges on that folder. You can perform an easy test by starting Spark-shell with the same user and try to write something to that folder, or if you have hdfs privilege check/grant.