Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Data is not loading into hdfs path by spark streaming.?

Highlighted

Data is not loading into hdfs path by spark streaming.?

New Contributor

Hi All,

I am trying to read twitter data into my hdfs path from kafka using spark streaming application.

Both my kafka and spark streaming apps are running fine, but spark app Is not able to write any data in to destination folder. somehow it is creating a Success file in the destination folder but not the actual data.

I' m using

Kafka 0.10.0.2.5 Spark 1.6.x.2.5 HDP2.5 version

Anyhelp is appreciated. I am Sharing my code.

Thanks.

spark-code.txt

1 REPLY 1
Highlighted

Re: Data is not loading into hdfs path by spark streaming.?

@shiva Yelpuri

Check that Spark user HDFS privileges on that folder. You can perform an easy test by starting Spark-shell with the same user and try to write something to that folder, or if you have hdfs privilege check/grant.

Don't have an account?
Coming from Hortonworks? Activate your account here