Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.5.0 - How to override default container log4j in YARN

SOLVED Go to solution
Highlighted

HDP 2.5.0 - How to override default container log4j in YARN

Contributor

In HDP 2.5 (managed using Ambari) how to override default container-log4j file. I am trying to add SocketAppender for all container logs so that it can be sent to logstash. In version HDP 2.4.0 I can find container-log4j file /usr/hdp/2.4.0.0-169/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties, but not in HDP 2.5. Any thoughts on how to configure container logs to include socket appender would be great.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HDP 2.5.0 - How to override default container log4j in YARN

Expert Contributor
@Saikiran Parepally

In HDP-2.5 you need to update the container-log4j.properties in the hadoop-yarn-server-nodemanager-2.7.3.2.5.X.Y-Z.jar file. First, locate this file under /usr/hdp/2.5..../...:

find /usr/hdp/ -name hadoop-yarn-server-nodemanager-*.jar

Then navigate to its directory.

You can get the current container-log4j.properties via this command:

jar xf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Edit the extracted file, e.g. add the SocketAppender you want. When you're finished, update the jar with your version:

jar uf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Obviously, this needs to be done on all the NodeManager nodes..

3 REPLIES 3

Re: HDP 2.5.0 - How to override default container log4j in YARN

Expert Contributor
@Saikiran Parepally

In HDP-2.5 you need to update the container-log4j.properties in the hadoop-yarn-server-nodemanager-2.7.3.2.5.X.Y-Z.jar file. First, locate this file under /usr/hdp/2.5..../...:

find /usr/hdp/ -name hadoop-yarn-server-nodemanager-*.jar

Then navigate to its directory.

You can get the current container-log4j.properties via this command:

jar xf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Edit the extracted file, e.g. add the SocketAppender you want. When you're finished, update the jar with your version:

jar uf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties

Obviously, this needs to be done on all the NodeManager nodes..

Re: HDP 2.5.0 - How to override default container log4j in YARN

Contributor

Thanks @gnovak. I was able to extract container-log4j.properties file as suggested and updated it to write to SocketAppender for container logs. But for some reason container is not writing logs to SocketAppender. If I add same log4j socketappender parameters to root logger from ambari, Nodemanager service is sending logs to socket appender. Any thoughts on how to configure container to write logs to socket appender using log4j?

Below is the updated container-log4j,

hadoop.root.logger=DEBUG,CLA,SA

log4j.appender.SA=org.apache.log4j.net.SocketAppender

log4j.appender.SA.Port=4560

log4j.appender.SA.RemoteHost=localhost

log4j.appender.SA.ReconnectionDelay=10000

log4j.appender.SA.Application=NM-${yarn.app.container.log.dir}

Looking at the conatiner launch script, I see below is being passed as root logger and it is getting overriden

-Dlog4j.configuration=container-log4j.properties -Dhadoop.root.logger=INFO,CLA

Re: HDP 2.5.0 - How to override default container log4j in YARN

New Contributor

Unable to update .jar file using command


  1. jar uf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties


Getting the following error.


java.io.IOException: Error in writing existing jar file

at sun.tools.jar.Main.run(Main.java:286)

at sun.tools.jar.Main.main(Main.java:1288)