Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark JavaRDD savaasTextFile

Highlighted

Spark JavaRDD savaasTextFile

Contributor

Hi,

I have JavaRDD, i want to save that into hdfs. am using fallowing code:

event1.saveAsTextFile("");

its creating folfer in hdfs but not writing anything into hdfs system and its not throwing any error too.

5 REPLIES 5
Highlighted

Re: Spark JavaRDD savaasTextFile

If you use Spark Streaming and the event RDD is empty then Spark will write a folder and not add any content.

see e.g. Deleting Directory in HDFS using Spark for how to avoid this.

If it is not Streaming, verify the JavaRDD is not empty

Highlighted

Re: Spark JavaRDD savaasTextFile

Contributor

@Bernhard Walter Its streaming only. but before writing contents into hdfs ,I printed out contents. its displaying content too

Highlighted

Re: Spark JavaRDD savaasTextFile

Contributor

I was able to solve the problem using coalease function.

Highlighted

Re: Spark JavaRDD savaasTextFile

Contributor

@vnandigam , are you running literally this code?

event1.saveAsTextFile("");

Because, the saveAsTextFile expects one argument to be passed, a path to the file, if you pass an empty string as path, you should not expect your rdd data to be saved into any file.

http://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaRDD.html#saveAsTextFile(j...

Highlighted

Re: Spark JavaRDD savaasTextFile

Contributor

@pbarna Hi, I removed path of my file here. I already passed destination follder

Don't have an account?
Coming from Hortonworks? Activate your account here