I would like to be able to do:
pyspark2 --master yarn --deploy-mode client
on Hadoop cluster.
It works if I use "hdfs:///" but also want to write to $HOME which is mounted on every node and is GPFS file system.
This works if I use
pyspark2 --master local
or if I use a standalone spark with
pyspark2 --master spark://...
but it does not work under yarn-local with file:///.