08-25-2017 03:55 AM
I am facing this issue in Cloudera 5.9. 5.11 and 5.12
when i am submitting a spark job through oozie this exception comes
/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied)
The spark project is built with scala 2.10.6 and java 1.8.
08-28-2017 12:03 AM
#1) Do you have spark gateway installed and client configuration deployed on this host?
#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?
Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir.
08-28-2017 03:15 AM
09-13-2017 06:50 AM
Yes through oozie in kerberised env, it shows Spark job as FAILED.
Application Manager tries two attempts and it fails
but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED