Thank you for the information. The need for dfs -du is to check how much disk space is available (before starting the job) and check how the job is generating data (how much data) ... View more

Thank you Steve for those insights . These are very helpful for beginners like me. ... View more

saveAsTextFile also does this job. I used following code . rdd.saveAsTextFile(outputFile, classOf[SnappyCodec]) ... View more