Created on 08-25-2017 03:55 AM - edited 09-16-2022 05:08 AM
I am facing this issue in Cloudera 5.9. 5.11 and 5.12
when i am submitting a spark job through oozie this exception comes
/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied)
The spark project is built with scala 2.10.6 and java 1.8.
Created 08-28-2017 12:03 AM
#1) Do you have spark gateway installed and client configuration deployed on this host?
#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?
Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir.
Created 09-13-2017 06:50 AM
Yes through oozie in kerberised env, it shows Spark job as FAILED.
Application Manager tries two attempts and it fails
but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED
Created on 04-10-2018 02:06 AM - edited 04-10-2018 02:08 AM
The easiest workaround for this is to copy the log4j.propeties file to oozie.sharelib.path. You can even add all the extra files and jars which are required in the yarn container when scheduling via oozie.
Created 04-10-2018 02:10 AM
The issue was related to customized environment and it is fixed now.
Created 01-29-2019 08:45 AM
Hello, I am facing the same problem, could you help me, and give me more detail, I will appreciate it.
Thanks in advance.