decrease log verbosity of container log output ?

[#1]

Hi,

I am facing lots of "bad dir status" issues on Nodemanagers while running huge job. This is caused by flooding the container log files with [DEBUG] messages (in detail the 'syslog' file is getting very large).

How can I modify the loglevel of the container specific log to reduce the size of the syslog file ?

E.g.:

$ ls -alh 
./container_e12_1453036276967_0004_01_000004:
total 7.4G
drwxr-s---  2 siad hadoop 4.0K Jan 17 18:39 .
drwxr-s--- 10 siad hadoop 4.0K Jan 17 20:21 ..
-rw-r-----  1 siad hadoop  222 Jan 17 18:47 stderr
-rw-r-----  1 siad hadoop    0 Jan 17 18:39 stdout
-rw-r-----  1 siad hadoop 7.4G Jan 17 18:47 syslog

In /etc/hadoop I cannot find any config setting including the value 'debug', so where does this DEBUG output come from ?

Thanks, Gerd

[#2]

@Gerd Koenig

See this yarn.app.mapreduce.am.log.level

[#3]

Hello @Neeraj Sabharwal ,

thanks for your hint, this property is set to

I checked the process list, and detected that the processes are using file 'container-log4j.properties' as log4j configuration.

The only file with that name I could find is:

/usr/hdp/2.2.4.2-2/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties

...and indeed, there is the property =>

hadoop.root.logger=DEBUG,CLA

Do I have to edit that file directly or is that just kind of template ?!?!

[#4]

@Gerd Koenig

You may want to see this and follow it LINK

[#5]

and, in addition, the process itself lists also the hadoop.root.logger property as DEBUG,CLA

Is that setting be applied from file /usr/hdp/2.2.4.2-2/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties, or from where ?!?!

[#6]

Is it set to debug in your case?

[#7]

Hi @Neeraj Sabharwal ,

nope, unfortunately not 😉

[#8]

@Gerd Koenig

Please see this...Not sure whats causing the inconsistencies in your env. You may want to restart the services

[root@phdns01 ~]# ps -ef | grep hadoop.root.logger

root 8571 26699 0 13:24 pts/2 00:00:00 grep hadoop.root.logger

Dhadoop.root.logger=INFO,RFA

[#9]

@Gerd Koenig has this been resolved? Can you provide your solution or accept the best answer?

[#10 · Accepted solution]

Seems like for that particular job the log setting got overwritten by a local container-log4j.properties, which included DEBUG loglevel.