How to configure spark-log4j-properties in Ambari UI?

fearghal_coyne — Tue, 21 Feb 2017 01:35:44 GMT

Hi,

I am trying to change the log4j properties in a Spark on Yarn application by changing the Advanced spark-log4j-properties config in Ambari UI. However, this does not seem to change the logs in any way. Right now logs are writing to both stderr and stdout and also there is no timestamp on the stdout log messages (which is needed). Is there another way that this should be done outside of the Ambari UI??

Spark version: 1.6 and HDP version: 2.5

Thanks,

Fearghal

Re: How to configure spark-log4j-properties in Ambari UI?

bikas — Tue, 21 Feb 2017 11:04:31 GMT

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.

question Re: How to configure spark-log4j-properties in Ambari UI? in Archives of Support Questions (Read Only)

How to configure spark-log4j-properties in Ambari UI?

Re: How to configure spark-log4j-properties in Ambari UI?