Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

decrease log verbosity of container log output ?

avatar
Guru

Hi,

I am facing lots of "bad dir status" issues on Nodemanagers while running huge job. This is caused by flooding the container log files with [DEBUG] messages (in detail the 'syslog' file is getting very large).

How can I modify the loglevel of the container specific log to reduce the size of the syslog file ?

E.g.:

$ ls -alh 
./container_e12_1453036276967_0004_01_000004:
total 7.4G
drwxr-s---  2 siad hadoop 4.0K Jan 17 18:39 .
drwxr-s--- 10 siad hadoop 4.0K Jan 17 20:21 ..
-rw-r-----  1 siad hadoop  222 Jan 17 18:47 stderr
-rw-r-----  1 siad hadoop    0 Jan 17 18:39 stdout
-rw-r-----  1 siad hadoop 7.4G Jan 17 18:47 syslog

In /etc/hadoop I cannot find any config setting including the value 'debug', so where does this DEBUG output come from ?

Thanks, Gerd

1 ACCEPTED SOLUTION

avatar
Guru

Seems like for that particular job the log setting got overwritten by a local container-log4j.properties, which included DEBUG loglevel.

View solution in original post

11 REPLIES 11

avatar
Master Mentor
@Gerd Koenig

See this yarn.app.mapreduce.am.log.level

avatar
Guru

Hello @Neeraj Sabharwal ,

thanks for your hint, this property is set to

1404-yarn-am-loglevel.png

I checked the process list, and detected that the processes are using file 'container-log4j.properties' as log4j configuration.

The only file with that name I could find is:

/usr/hdp/2.2.4.2-2/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties

...and indeed, there is the property =>

hadoop.root.logger=DEBUG,CLA

Do I have to edit that file directly or is that just kind of template ?!?!

avatar
Master Mentor
@Gerd Koenig

You may want to see this and follow it LINK

avatar
Guru

and, in addition, the process itself lists also the hadoop.root.logger property as DEBUG,CLA

1405-hadooprootlogger.png

Is that setting be applied from file /usr/hdp/2.2.4.2-2/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties, or from where ?!?!

avatar
Master Mentor

1415-screen-shot-2016-01-17-at-25123-pm.png

Is it set to debug in your case?

avatar
Guru

Hi @Neeraj Sabharwal ,

nope, unfortunately not 😉

1406-hadooprootlogger-in-ambari.png

avatar
Master Mentor

@Gerd Koenig

Please see this...Not sure whats causing the inconsistencies in your env. You may want to restart the services

[root@phdns01 ~]# ps -ef | grep hadoop.root.logger

root 8571 26699 0 13:24 pts/2 00:00:00 grep hadoop.root.logger

Dhadoop.root.logger=INFO,RFA

1419-screen-shot-2016-01-17-at-42403-pm.png

avatar
Master Mentor

@Gerd Koenig has this been resolved? Can you provide your solution or accept the best answer?

avatar
Guru

Seems like for that particular job the log setting got overwritten by a local container-log4j.properties, which included DEBUG loglevel.