Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Log4j.properties permission denied

Solved Go to solution

Log4j.properties permission denied

Contributor

 

 

Dear All,

 

I am facing this issue in Cloudera 5.9. 5.11 and 5.12

 

when i am submitting a spark job through oozie this exception comes

 

 

/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied)

 

 

The spark project is built with scala 2.10.6 and java 1.8.

 

regards

Shafi

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Log4j.properties permission denied

Master Guru
If you're seeing this exception in the Oozie Spark action launcher log,
please ignore it as it may be expected in a secure environment. If your
launcher/action is truly failing, then the real exception will lie in the
logs that follow/other parts of the log - the log4j permission error can be
treated as a red herring as you investigate.
6 REPLIES 6

Re: Log4j.properties permission denied

Contributor

#1) Do you have spark gateway installed and client configuration deployed on this host?

#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?

 

 

Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir.

Re: Log4j.properties permission denied

Master Guru
If you're seeing this exception in the Oozie Spark action launcher log,
please ignore it as it may be expected in a secure environment. If your
launcher/action is truly failing, then the real exception will lie in the
logs that follow/other parts of the log - the log4j permission error can be
treated as a red herring as you investigate.

Re: Log4j.properties permission denied

Contributor

Hi

 

Yes through oozie in kerberised env, it shows Spark job as FAILED.

 

Application Manager tries two attempts and it fails

 

but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED

 

regards

Shafi

Re: Log4j.properties permission denied

New Contributor

The easiest workaround for this is to copy the log4j.propeties file to oozie.sharelib.path. You can even add all the extra files and jars which are required in the yarn container when scheduling via oozie.

Re: Log4j.properties permission denied

Contributor

Hi

 

The issue was related to customized environment and it is fixed now.

 

regards

Shafi

Re: Log4j.properties permission denied

New Contributor

Hello, I am facing the same problem, could you help me, and give me more detail, I will appreciate it.

 

Thanks in advance.