I am facing this issue in Cloudera 5.9. 5.11 and 5.12
when i am submitting a spark job through oozie this exception comes
/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied)
The spark project is built with scala 2.10.6 and java 1.8.
#1) Do you have spark gateway installed and client configuration deployed on this host?
#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?
Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir.
Yes through oozie in kerberised env, it shows Spark job as FAILED.
Application Manager tries two attempts and it fails
but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED
The easiest workaround for this is to copy the log4j.propeties file to oozie.sharelib.path. You can even add all the extra files and jars which are required in the yarn container when scheduling via oozie.