Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to configure spark-log4j-properties in Ambari UI?

avatar
New Contributor

Hi,

I am trying to change the log4j properties in a Spark on Yarn application by changing the Advanced spark-log4j-properties config in Ambari UI. However, this does not seem to change the logs in any way. Right now logs are writing to both stderr and stdout and also there is no timestamp on the stdout log messages (which is needed). Is there another way that this should be done outside of the Ambari UI??

Spark version: 1.6 and HDP version: 2.5

Thanks,

Fearghal

1 ACCEPTED SOLUTION

avatar
Super Collaborator

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.

View solution in original post

1 REPLY 1

avatar
Super Collaborator

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.