Posts: 30
Registered: ‎03-02-2017 permission denied



Dear All,


I am facing this issue in Cloudera 5.9. 5.11 and 5.12


when i am submitting a spark job through oozie this exception comes



/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/ (permission denied)



The spark project is built with scala 2.10.6 and java 1.8.




Cloudera Employee
Posts: 10
Registered: ‎03-01-2016

Re: permission denied

#1) Do you have spark gateway installed and client configuration deployed on this host?

#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?



Usgually the client should be able to source the log4j file from /etc/spark/conf/, instead of the NodeManager process dir.

Posts: 1,606
Kudos: 300
Solutions: 243
Registered: ‎07-31-2013

Re: permission denied

If you're seeing this exception in the Oozie Spark action launcher log,
please ignore it as it may be expected in a secure environment. If your
launcher/action is truly failing, then the real exception will lie in the
logs that follow/other parts of the log - the log4j permission error can be
treated as a red herring as you investigate.
Posts: 30
Registered: ‎03-02-2017

Re: permission denied



Yes through oozie in kerberised env, it shows Spark job as FAILED.


Application Manager tries two attempts and it fails


but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED