Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to configure spark-log4j-properties in Ambari UI?

Solved Go to solution

How to configure spark-log4j-properties in Ambari UI?

New Contributor

Hi,

I am trying to change the log4j properties in a Spark on Yarn application by changing the Advanced spark-log4j-properties config in Ambari UI. However, this does not seem to change the logs in any way. Right now logs are writing to both stderr and stdout and also there is no timestamp on the stdout log messages (which is needed). Is there another way that this should be done outside of the Ambari UI??

Spark version: 1.6 and HDP version: 2.5

Thanks,

Fearghal

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to configure spark-log4j-properties in Ambari UI?

Expert Contributor

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.

1 REPLY 1

Re: How to configure spark-log4j-properties in Ambari UI?

Expert Contributor

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.

Don't have an account?
Coming from Hortonworks? Activate your account here