could you please post little more information on the job, the submit command etc. What is your data source? ... View more

I guess, if the data set does not contain a '\t' char then '\t'.join and saveAsTextFile should work for you. Else, you just need to wrap the strings within " as with normal CSVs. ... View more

Could you provide more details on the your RDD that you would like to save tab delimited? On the question about storing the DataFrames as a tab delimited file, below is what I have in scala using the package spark-csv df.write.format("com.databricks.spark.csv").option("delimiter", "\t").save("output path") EDIT With the RDD of tuples, as you mentioned, either you could join by "\t" on the tuple or use mkString if you prefer not to use an additional library. On your RDD of tuple you could do something like .map { x =>x.productIterator.mkString("\t") }.saveAsTextFile("path-to-store") @Don Jernigan ... View more

Is your RDD an RDD of strings? On the second part of the question, if you are using the spark-csv, the package supports saving simple (non-nested) DataFrame. There is an option to specify the delimiter which is , by default but can be changed. eg - .save('filename.csv', 'com.databricks.spark.csv',delimiter="DELIM") ... View more

Difference is noticeable only when we run it in a cluster mode without actually knowing where the driver is. On the other case, if we know where the driver is set to launch, both methods are similar in action. --files is a submit time parameter, main() can run anywhere and just need to know the file name. In code, I can refer to the file by a file:// call. In case of addFile(), since this is a code level setting, the main() need to know the file location in order to perform the add() . As per the API doc, The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. ... View more

Thank You @clukasik and @Jitendra Yadav. Appreciate your help. ... View more

@Benjamin Leonhardi. Thanks for pointing this out. I over looked this flag. ... View more

@Rajkumar Singh, don't the application.properties.file need to be in a key value format? ... View more

Thanks @Jitendra Yadav. I will take a look at the addFile API. I would like to try getting control on the driver as clukasik pointed out. ... View more

@clukasik, Thank You, I have had a look at broadcast variables. But I guess with the current requirement, I just require the RDD. ... View more