Support Questions
Find answers, ask questions, and share your expertise

Spark JavaRDD savaasTextFile

Contributor

Hi,

I have JavaRDD, i want to save that into hdfs. am using fallowing code:

event1.saveAsTextFile("");

its creating folfer in hdfs but not writing anything into hdfs system and its not throwing any error too.

5 REPLIES 5

If you use Spark Streaming and the event RDD is empty then Spark will write a folder and not add any content.

see e.g. Deleting Directory in HDFS using Spark for how to avoid this.

If it is not Streaming, verify the JavaRDD is not empty

Contributor

@Bernhard Walter Its streaming only. but before writing contents into hdfs ,I printed out contents. its displaying content too

Contributor

I was able to solve the problem using coalease function.

Contributor

@vnandigam , are you running literally this code?

event1.saveAsTextFile("");

Because, the saveAsTextFile expects one argument to be passed, a path to the file, if you pass an empty string as path, you should not expect your rdd data to be saved into any file.

http://spark.apache.org/docs/latest/api/java/org/apache/spark/api/java/JavaRDD.html#saveAsTextFile(j...

Contributor

@pbarna Hi, I removed path of my file here. I already passed destination follder

; ;