Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Log4j.properties permission denied

avatar
Contributor

 

 

Dear All,

 

I am facing this issue in Cloudera 5.9. 5.11 and 5.12

 

when i am submitting a spark job through oozie this exception comes

 

 

/var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied)

 

 

The spark project is built with scala 2.10.6 and java 1.8.

 

regards

Shafi

1 ACCEPTED SOLUTION

avatar
Mentor
If you're seeing this exception in the Oozie Spark action launcher log,
please ignore it as it may be expected in a secure environment. If your
launcher/action is truly failing, then the real exception will lie in the
logs that follow/other parts of the log - the log4j permission error can be
treated as a red herring as you investigate.

View solution in original post

6 REPLIES 6

avatar
Expert Contributor

#1) Do you have spark gateway installed and client configuration deployed on this host?

#2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?

 

 

Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir.

avatar
Mentor
If you're seeing this exception in the Oozie Spark action launcher log,
please ignore it as it may be expected in a secure environment. If your
launcher/action is truly failing, then the real exception will lie in the
logs that follow/other parts of the log - the log4j permission error can be
treated as a red herring as you investigate.

avatar
Contributor

Hi

 

Yes through oozie in kerberised env, it shows Spark job as FAILED.

 

Application Manager tries two attempts and it fails

 

but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED

 

regards

Shafi

avatar
Explorer

The easiest workaround for this is to copy the log4j.propeties file to oozie.sharelib.path. You can even add all the extra files and jars which are required in the yarn container when scheduling via oozie.

avatar
Contributor

Hi

 

The issue was related to customized environment and it is fixed now.

 

regards

Shafi

avatar
New Contributor

Hello, I am facing the same problem, could you help me, and give me more detail, I will appreciate it.

 

Thanks in advance.