Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

How to configure spark-log4j-properties in Ambari UI?

New Contributor

Hi,

I am trying to change the log4j properties in a Spark on Yarn application by changing the Advanced spark-log4j-properties config in Ambari UI. However, this does not seem to change the logs in any way. Right now logs are writing to both stderr and stdout and also there is no timestamp on the stdout log messages (which is needed). Is there another way that this should be done outside of the Ambari UI??

Spark version: 1.6 and HDP version: 2.5

Thanks,

Fearghal

1 ACCEPTED SOLUTION

Expert Contributor

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.

View solution in original post

1 REPLY 1

Expert Contributor

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.