08-25-2017 03:55 AM
I am facing this issue in Cloudera 5.9. 5.11 and 5.12
when i am submitting a spark job through oozie this exception comes
/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied)
The spark project is built with scala 2.10.6 and java 1.8.
08-28-2017 12:03 AM
#1) Do you have spark gateway installed and client configuration deployed on this host?
#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?
Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir.
08-28-2017 03:15 AM
09-13-2017 06:50 AM
Yes through oozie in kerberised env, it shows Spark job as FAILED.
Application Manager tries two attempts and it fails
but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED
04-10-2018 02:06 AM - edited 04-10-2018 02:08 AM
The easiest workaround for this is to copy the log4j.propeties file to oozie.sharelib.path. You can even add all the extra files and jars which are required in the yarn container when scheduling via oozie.