question Re: decrease log verbosity of container log output ? in Archives of Support Questions (Read Only)

decrease log verbosity of container log output ?

geko — Mon, 18 Jan 2016 03:25:11 GMT

Hi,

I am facing lots of "bad dir status" issues on Nodemanagers while running huge job. This is caused by flooding the container log files with [DEBUG] messages (in detail the 'syslog' file is getting very large).

How can I modify the loglevel of the container specific log to reduce the size of the syslog file ?

E.g.:

$ ls -alh 
./container_e12_1453036276967_0004_01_000004:
total 7.4G
drwxr-s---  2 siad hadoop 4.0K Jan 17 18:39 .
drwxr-s--- 10 siad hadoop 4.0K Jan 17 20:21 ..
-rw-r-----  1 siad hadoop  222 Jan 17 18:47 stderr
-rw-r-----  1 siad hadoop    0 Jan 17 18:39 stdout
-rw-r-----  1 siad hadoop 7.4G Jan 17 18:47 syslog

In /etc/hadoop I cannot find any config setting including the value 'debug', so where does this DEBUG output come from ?

Thanks, Gerd

Re: decrease log verbosity of container log output ?

nsabharwal — Mon, 18 Jan 2016 03:33:58 GMT

@Gerd Koenig

See this yarn.app.mapreduce.am.log.level

Re: decrease log verbosity of container log output ?

geko — Mon, 19 Aug 2019 12:11:28 GMT

Hello @Neeraj Sabharwal ,

thanks for your hint, this property is set to

I checked the process list, and detected that the processes are using file 'container-log4j.properties' as log4j configuration.

The only file with that name I could find is:

/usr/hdp/2.2.4.2-2/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties

...and indeed, there is the property =>

hadoop.root.logger=DEBUG,CLA

Do I have to edit that file directly or is that just kind of template ?!?!

Re: decrease log verbosity of container log output ?

nsabharwal — Mon, 18 Jan 2016 03:44:22 GMT

@Gerd Koenig

You may want to see this and follow it LINK

Re: decrease log verbosity of container log output ?

geko — Mon, 19 Aug 2019 12:11:19 GMT

and, in addition, the process itself lists also the hadoop.root.logger property as DEBUG,CLA

Is that setting be applied from file /usr/hdp/2.2.4.2-2/hadoop/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties, or from where ?!?!

Re: decrease log verbosity of container log output ?

nsabharwal — Mon, 19 Aug 2019 12:11:11 GMT

Is it set to debug in your case?

Re: decrease log verbosity of container log output ?

geko — Mon, 19 Aug 2019 12:11:03 GMT

Hi @Neeraj Sabharwal ,

nope, unfortunately not 😉

Re: decrease log verbosity of container log output ?

nsabharwal — Mon, 19 Aug 2019 12:10:55 GMT

@Gerd Koenig

Please see this...Not sure whats causing the inconsistencies in your env. You may want to restart the services

[root@phdns01 ~]# ps -ef | grep hadoop.root.logger

root 8571 26699 0 13:24 pts/2 00:00:00 grep hadoop.root.logger

Dhadoop.root.logger=INFO,RFA

Re: decrease log verbosity of container log output ?

aervits — Wed, 03 Feb 2016 09:48:36 GMT

@Gerd Koenig has this been resolved? Can you provide your solution or accept the best answer?

Re: decrease log verbosity of container log output ?

geko — Fri, 05 Feb 2016 01:06:08 GMT

Seems like for that particular job the log setting got overwritten by a local container-log4j.properties, which included DEBUG loglevel.

Re: decrease log verbosity of container log output ?

geko — Fri, 05 Feb 2016 01:06:42 GMT

@All, sorry for any inconvenience 😉

Re: decrease log verbosity of container log output ?

aervits — Fri, 05 Feb 2016 01:09:59 GMT

@Gerd Koenig good job, re-accepted to give you credit