Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Writing to $HOME from pyspark2 --master yarn --deploy-mode client


Writing to $HOME from pyspark2 --master yarn --deploy-mode client

Hi All,


I would like to be able to do:



pyspark2 --master yarn --deploy-mode client

on Hadoop cluster.

It works if I use "hdfs:///" but also want to write to $HOME which is mounted on every node and is GPFS file system.



This works if I use

pyspark2 --master local

or if I use a standalone spark with

pyspark2 --master spark://...

but it does not work under yarn-local with file:///.


Thank you,




Don't have an account?
Coming from Hortonworks? Activate your account here