Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

write a file to HDFS using Spark

Solved Go to solution

write a file to HDFS using Spark

New Contributor

hi,

i am able to read a file from HDFS in Spark e using sc.textFile("path")

Can smeone help me How to write a file in HDFS using spark

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: write a file to HDFS using Spark

Explorer

Hi,

 

According documentation, you may try the saveAsTextFile method:

http://spark.apache.org/docs/latest/programming-guide.html

saveAsTextFile(path)Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in the file.
2 REPLIES 2
Highlighted

Re: write a file to HDFS using Spark

Explorer

Hi,

 

According documentation, you may try the saveAsTextFile method:

http://spark.apache.org/docs/latest/programming-guide.html

saveAsTextFile(path)Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in the file.

Re: write a file to HDFS using Spark

New Contributor

Thanks