How to configure spark-log4j-properties in Ambari UI?

[#1]

Hi,

I am trying to change the log4j properties in a Spark on Yarn application by changing the Advanced spark-log4j-properties config in Ambari UI. However, this does not seem to change the logs in any way. Right now logs are writing to both stderr and stdout and also there is no timestamp on the stdout log messages (which is needed). Is there another way that this should be done outside of the Ambari UI??

Spark version: 1.6 and HDP version: 2.5

Thanks,

Fearghal

[#2 · Accepted solution]

That log4j only affects the service daemons like spark history server and anything you run on the client machines. For executors/drivers that run on YARN machines, the log4j file has to be passed to them using "--files" option during job submit and then referenced via JVM property via JVM arguments "-Dlog4j.configuration" . See here for examples.