Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

write a file to HDFS using Spark

SOLVED Go to solution

write a file to HDFS using Spark

New Contributor

hi,

i am able to read a file from HDFS in Spark e using sc.textFile("path")

Can smeone help me How to write a file in HDFS using spark

1 ACCEPTED SOLUTION

Accepted Solutions

Re: write a file to HDFS using Spark

Explorer

Hi,

 

According documentation, you may try the saveAsTextFile method:

http://spark.apache.org/docs/latest/programming-guide.html

saveAsTextFile(path)Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in the file.
2 REPLIES 2

Re: write a file to HDFS using Spark

Explorer

Hi,

 

According documentation, you may try the saveAsTextFile method:

http://spark.apache.org/docs/latest/programming-guide.html

saveAsTextFile(path)Write the elements of the dataset as a text file (or set of text files) in a given directory in the local filesystem, HDFS or any other Hadoop-supported file system. Spark will call toString on each element to convert it to a line of text in the file.

Re: write a file to HDFS using Spark

New Contributor

Thanks