Support Questions
Find answers, ask questions, and share your expertise

Spark JavaRDD savaasTextFile

Spark JavaRDD savaasTextFile

Contributor

Hi,

I have JavaRDD, i want to save that into hdfs. am using fallowing code:

event1.saveAsTextFile("");

its creating folfer in hdfs but not writing anything into hdfs system and its not throwing any error too.

5 REPLIES 5

Re: Spark JavaRDD savaasTextFile

If you use Spark Streaming and the event RDD is empty then Spark will write a folder and not add any content.

see e.g. Deleting Directory in HDFS using Spark for how to avoid this.

If it is not Streaming, verify the JavaRDD is not empty

Re: Spark JavaRDD savaasTextFile

Contributor

@Bernhard Walter Its streaming only. but before writing contents into hdfs ,I printed out contents. its displaying content too

Re: Spark JavaRDD savaasTextFile

Contributor

I was able to solve the problem using coalease function.

Re: Spark JavaRDD savaasTextFile

Contributor

@vnandigam , are you running literally this code?

event1.saveAsTextFile("");

Because, the saveAsTextFile expects one argument to be passed, a path to the file, if you pass an empty string as path, you should not expect your rdd data to be saved into any file.

http://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaRDD.html#saveAsTextFile(j...

Re: Spark JavaRDD savaasTextFile

Contributor

@pbarna Hi, I removed path of my file here. I already passed destination follder