HDP 2.5.0 - How to override default container log4j in YARN

[#1]

In HDP 2.5 (managed using Ambari) how to override default container-log4j file. I am trying to add SocketAppender for all container logs so that it can be sent to logstash. In version HDP 2.4.0 I can find container-log4j file /usr/hdp/2.4.0.0-169/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties, but not in HDP 2.5. Any thoughts on how to configure container logs to include socket appender would be great.

[#2 · Accepted solution]

@Saikiran Parepally

In HDP-2.5 you need to update the container-log4j.properties in the hadoop-yarn-server-nodemanager-2.7.3.2.5.X.Y-Z.jar file. First, locate this file under /usr/hdp/2.5..../...:

find /usr/hdp/ -name hadoop-yarn-server-nodemanager-*.jar

Then navigate to its directory.

You can get the current container-log4j.properties via this command:

jar xf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Edit the extracted file, e.g. add the SocketAppender you want. When you're finished, update the jar with your version:

jar uf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Obviously, this needs to be done on all the NodeManager nodes..

[#3]

Thanks @gnovak. I was able to extract container-log4j.properties file as suggested and updated it to write to SocketAppender for container logs. But for some reason container is not writing logs to SocketAppender. If I add same log4j socketappender parameters to root logger from ambari, Nodemanager service is sending logs to socket appender. Any thoughts on how to configure container to write logs to socket appender using log4j?

Below is the updated container-log4j,

hadoop.root.logger=DEBUG,CLA,SA

log4j.appender.SA=org.apache.log4j.net.SocketAppender

log4j.appender.SA.Port=4560

log4j.appender.SA.RemoteHost=localhost

log4j.appender.SA.ReconnectionDelay=10000

log4j.appender.SA.Application=NM-${yarn.app.container.log.dir}

Looking at the conatiner launch script, I see below is being passed as root logger and it is getting overriden

-Dlog4j.configuration=container-log4j.properties -Dhadoop.root.logger=INFO,CLA

[#4]

Unable to update .jar file using command

1. jar uf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Getting the following error.

java.io.IOException: Error in writing existing jar file

at sun.tools.jar.Main.run(Main.java:286)

at sun.tools.jar.Main.main(Main.java:1288)