question Re: How do you write a RDD as a tab delimited file in pyspark? in Archives of Support Questions (Read Only)

How do you write a RDD as a tab delimited file in pyspark?

don_jernigan — Mon, 19 Sep 2022 18:45:28 GMT

I have a RDD I'd like to write as tab delimited. I also want to write a data frame as tab delimited. How do I do this?

Re: How do you write a RDD as a tab delimited file in pyspark?

arunak — Thu, 14 Jul 2016 03:58:21 GMT

Is your RDD an RDD of strings?

On the second part of the question, if you are using the spark-csv, the package supports saving simple (non-nested) DataFrame. There is an option to specify the delimiter which is , by default but can be changed.

eg - .save('filename.csv', 'com.databricks.spark.csv',delimiter="DELIM")

Re: How do you write a RDD as a tab delimited file in pyspark?

sunile_manjee — Thu, 14 Jul 2016 10:24:58 GMT

@Binu Mathew do you have any thoughts?

Re: How do you write a RDD as a tab delimited file in pyspark?

arunak — Thu, 14 Jul 2016 21:23:02 GMT

Could you provide more details on the your RDD that you would like to save tab delimited? On the question about storing the DataFrames as a tab delimited file, below is what I have in scala using the package spark-csv

df.write.format("com.databricks.spark.csv").option("delimiter", "\t").save("output path")

EDIT With the RDD of tuples, as you mentioned, either you could join by "\t" on the tuple or use mkString if you prefer not to use an additional library. On your RDD of tuple you could do something like

.map { x =>x.productIterator.mkString("\t") }.saveAsTextFile("path-to-store")

@Don Jernigan

Re: How do you write a RDD as a tab delimited file in pyspark?

don_jernigan — Thu, 14 Jul 2016 23:04:12 GMT

Essentially I have a python tuple ('a','b','c','x','y','z') that are all strings. I could just map them into a single concatenation of ('a\tb\tc\tx\ty\tz'), then saveAsTextFile(path). But I was wondering if there was a better way than using an external package which could just be encapsulating that .map(lambda x: "\t'".join(x) ).

Re: How do you write a RDD as a tab delimited file in pyspark?

arunak — Fri, 15 Jul 2016 01:48:18 GMT

I guess, if the data set does not contain a '\t' char then '\t'.join and saveAsTextFile should work for you. Else, you just need to wrap the strings within " as with normal CSVs.

Re: How do you write a RDD as a tab delimited file in pyspark?

doug_mengistu — Mon, 18 Jul 2016 22:42:41 GMT

Try this, but this version is for version 1.5 and up

data.write.format('com.databricks.spark.csv').options(delimiter="\t", codec="org.apache.hadoop.io.compress.GzipCodec").save('s3a://myBucket/myPath')