Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Writing to $HOME from pyspark2 --master yarn --deploy-mode client

Highlighted

Writing to $HOME from pyspark2 --master yarn --deploy-mode client

Hi All,

 

I would like to be able to do:

ts.saveAsTextFile("file:///home/me/my")

from

pyspark2 --master yarn --deploy-mode client

on Hadoop cluster.

It works if I use "hdfs:///" but also want to write to $HOME which is mounted on every node and is GPFS file system.

 

 

This works if I use

pyspark2 --master local

or if I use a standalone spark with

pyspark2 --master spark://...

but it does not work under yarn-local with file:///.

 

Thank you,

Igor

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here