how to configure the log4j in order to rotate the rm-audit files

mike_bronson7 — Mon, 27 Aug 2018 21:57:13 GMT

hi all,

Under /var/log/hadoop-yarn/yarn , we have the following files ( pic 1) :

how to reconfigure the Advanced yarn-log4j ( under YARN --> CONFIG ) , in order to rotate the files

so max backup files will be - 10 , and max file size will limit to 10K

PIC 1

ls -ltr
total 2023884
-rw-r--r--. 1 yarn hadoop       274 Aug  7  2017 rm-audit.log.2017-08-07
-rw-r--r--. 1 yarn hadoop      1644 Aug  9  2017 rm-audit.log.2017-08-09
-rw-r--r--. 1 yarn hadoop      4346 Aug 10  2017 rm-audit.log.2017-08-10
-rw-r--r--. 1 yarn hadoop      6450 Aug 15  2017 rm-audit.log.2017-08-15
-rw-r--r--. 1 yarn hadoop       824 Aug 16  2017 rm-audit.log.2017-08-16
-rw-r--r--. 1 yarn hadoop     65573 Aug 17  2017 rm-audit.log.2017-08-17
-rw-r--r--. 1 yarn hadoop       274 Sep  6  2017 rm-audit.log.2017-09-06
-rw-r--r--. 1 yarn hadoop       274 Sep 24  2017 rm-audit.log.2017-09-24
-rw-r--r--  1 yarn hadoop       822 Sep 27  2017 rm-audit.log.2017-09-27
-rw-r--r--  1 yarn hadoop    135508 Sep 28  2017 rm-audit.log.2017-09-28
-rw-r--r--  1 yarn hadoop      3838 Oct  1  2017 rm-audit.log.2017-10-01
-rw-r--r--  1 yarn hadoop    175181 Oct  2  2017 rm-audit.log.2017-10-02
-rw-r--r--  1 yarn hadoop     54213 Oct  3  2017 rm-audit.log.2017-10-03
-rw-r--r--  1 yarn hadoop      5313 Oct  8  2017 rm-audit.log.2017-10-08
-rw-r--r--  1 yarn hadoop     32125 Oct  9  2017 rm-audit.log.2017-10-09
-rw-r--r--  1 yarn hadoop     25134 Oct 10  2017 rm-audit.log.2017-10-10
-rw-r--r--  1 yarn hadoop     89328 Oct 11  2017 rm-audit.log.2017-10-11
-rw-r--r--  1 yarn hadoop     47737 Oct 15  2017 rm-audit.log.2017-10-15
-rw-r--r--  1 yarn hadoop     14218 Oct 16  2017 rm-audit.log.2017-10-16
-rw-r--r--  1 yarn hadoop    126010 Oct 17  2017 rm-audit.log.2017-10-17
-rw-r--r--  1 yarn hadoop     89089 Oct 18  2017 rm-audit.log.2017-10-18
-rw-r--r--  1 yarn hadoop    157802 Oct 19  2017 rm-audit.log.2017-10-19
-rw-r--r--  1 yarn hadoop     23173 Oct 20  2017 rm-audit.log.2017-10-20
-rw-r--r--  1 yarn hadoop     25958 Oct 21  2017 rm-audit.log.2017-10-21
-rw-r--r--  1 yarn hadoop     22483 Oct 22  2017 rm-audit.log.2017-10-22
-rw-r--r--  1 yarn hadoop    153963 Oct 23  2017 rm-audit.log.2017-10-23
-rw-r--r--  1 yarn hadoop     82562 Oct 24  2017 rm-audit.log.2017-10-24
-rw-r--r--  1 yarn hadoop       274 Oct 29  2017 rm-audit.log.2017-10-29
-rw-r--r--  1 yarn hadoop   2139002 Oct 30  2017 rm-audit.log.2017-10-30
-rw-r--r--  1 yarn hadoop    969480 Oct 31  2017 rm-audit.log.2017-10-31
-rw-r--r--  1 yarn hadoop    991997 Nov  1  2017 rm-audit.log.2017-11-01
-rw-r--r--  1 yarn hadoop    994624 Nov  2  2017 rm-audit.log.2017-11-02
-rw-r--r--  1 yarn hadoop    978260 Nov  3  2017 rm-audit.log.2017-11-03
-rw-r--r--  1 yarn hadoop    977769 Nov  4  2017 rm-audit.log.2017-11-04
-rw-r--r--  1 yarn hadoop   1425299 Nov  5  2017 rm-audit.log.2017-11-05
-rw-r--r--  1 yarn hadoop   1499223 Nov  6  2017 rm-audit.log.2017-11-06
-rw-r--r--  1 yarn hadoop   1535388 Nov  7  2017 rm-audit.log.2017-11-07
-rw-r--r--  1 yarn hadoop       274 Nov 21  2017 rm-audit.log.2017-11-21
-rw-r--r--  1 yarn hadoop   4197197 Nov 27  2017 rm-audit.log.2017-11-27
-rw-r--r--  1 yarn hadoop   1145099 Nov 28  2017 rm-audit.log.2017-11-28
-rw-r--r--  1 yarn hadoop    802082 Nov 29  2017 rm-audit.log.2017-11-29
-rw-r--r--  1 yarn hadoop    623300 Nov 30  2017 rm-audit.log.2017-11-30
-rw-r--r--  1 yarn hadoop    636433 Dec  1  2017 rm-audit.log.2017-12-01
-rw-r--r--  1 yarn hadoop    798816 Dec  2  2017 rm-audit.log.2017-12-02
-rw-r--r--  1 yarn hadoop   1040673 Dec  3  2017 rm-audit.log.2017-12-03
-rw-r--r--  1 yarn hadoop    835143 Dec  4  2017 rm-audit.log.2017-12-04
-rw-r--r--  1 yarn hadoop   4000819 Dec  5  2017 rm-audit.log.2017-12-05
-rw-r--r--  1 yarn hadoop  33406088 Dec  6  2017 rm-audit.log.2017-12-06
-rw-r--r--  1 yarn hadoop  21565330 Dec  7  2017 rm-audit.log.2017-12-07
-rw-r--r--  1 yarn hadoop       274 Dec 11  2017 rm-audit.log.2017-12-11
-rw-r--r--  1 yarn hadoop       191 Dec 12  2017 rm-audit.log.2017-12-12
-rw-r--r--  1 yarn hadoop       274 Dec 14  2017 rm-audit.log.2017-12-14
-rw-r--r--  1 yarn hadoop       548 Dec 17  2017 rm-audit.log.2017-12-17
-rw-r--r--  1 yarn hadoop      2192 Dec 19  2017 rm-audit.log.2017-12-19
-rw-r--r--  1 yarn hadoop       274 Dec 27  2017 rm-audit.log.2017-12-27
-rw-r--r--  1 yarn hadoop   2754580 Dec 31  2017 rm-audit.log.2017-12-31
-rw-r--r--  1 yarn hadoop   1951645 Jan  1  2018 rm-audit.log.2018-01-01
-rw-r--r--  1 yarn hadoop       548 Jan  3  2018 rm-audit.log.2018-01-03
-rw-r--r--  1 yarn hadoop       274 Jan  8  2018 rm-audit.log.2018-01-08
-rw-r--r--  1 yarn hadoop       274 Jan 10  2018 rm-audit.log.2018-01-10
-rw-r--r--  1 yarn hadoop   2145304 Jan 11  2018 rm-audit.log.2018-01-11
-rw-r--r--  1 yarn hadoop    197143 Jan 13  2018 rm-audit.log.2018-01-13
-rw-r--r--  1 yarn hadoop      1370 Jan 14  2018 rm-audit.log.2018-01-14
-rw-r--r--  1 yarn hadoop       274 Jan 15  2018 rm-audit.log.2018-01-15
-rw-r--r--  1 yarn hadoop       274 Jan 18  2018 rm-audit.log.2018-01-18
-rw-r--r--  1 yarn hadoop   2363585 Jan 19  2018 rm-audit.log.2018-01-19
-rw-r--r--  1 yarn hadoop       822 Jan 20  2018 rm-audit.log.2018-01-20
-rw-r--r--  1 yarn hadoop   7395951 Jan 21  2018 rm-audit.log.2018-01-21
-rw-r--r--  1 yarn hadoop    241166 Jan 22  2018 rm-audit.log.2018-01-22
-rw-r--r--  1 yarn hadoop     19976 Jan 23  2018 rm-audit.log.2018-01-23
-rw-r--r--  1 yarn hadoop  10999019 Jan 24  2018 rm-audit.log.2018-01-24
-rw-r--r--  1 yarn hadoop  17312694 Jan 25  2018 rm-audit.log.2018-01-25
-rw-r--r--  1 yarn hadoop  31321833 Jan 26  2018 rm-audit.log.2018-01-26
-rw-r--r--  1 yarn hadoop  21793442 Jan 27  2018 rm-audit.log.2018-01-27
-rw-r--r--  1 yarn hadoop  28805081 Jan 28  2018 rm-audit.log.2018-01-28
-rw-r--r--  1 yarn hadoop  32190177 Jan 29  2018 rm-audit.log.2018-01-29
-rw-r--r--  1 yarn hadoop  21629686 Jan 30  2018 rm-audit.log.2018-01-30
-rw-r--r--  1 yarn hadoop       274 Feb  1  2018 rm-audit.log.2018-02-01
-rw-r--r--  1 yarn hadoop  15550664 Feb  4  2018 rm-audit.log.2018-02-04
-rw-r--r--  1 yarn hadoop       548 Feb  6  2018 rm-audit.log.2018-02-06
-rw-r--r--  1 yarn hadoop  12077408 Feb 13  2018 rm-audit.log.2018-02-13
-rw-r--r--  1 yarn hadoop    403786 Feb 14  2018 rm-audit.log.2018-02-14
-rw-r--r--  1 yarn hadoop    421911 Feb 15  2018 rm-audit.log.2018-02-15
-rw-r--r--  1 yarn hadoop    202958 Feb 16  2018 rm-audit.log.2018-02-16
-rw-r--r--  1 yarn hadoop    208080 Feb 17  2018 rm-audit.log.2018-02-17
-rw-r--r--  1 yarn hadoop    485789 Feb 18  2018 rm-audit.log.2018-02-18
-rw-r--r--  1 yarn hadoop    753580 Feb 19  2018 rm-audit.log.2018-02-19
-rw-r--r--  1 yarn hadoop    389283 Feb 20  2018 rm-audit.log.2018-02-20
-rw-r--r--  1 yarn hadoop    403405 Feb 21  2018 rm-audit.log.2018-02-21
-rw-r--r--  1 yarn hadoop    475882 Feb 22  2018 rm-audit.log.2018-02-22
-rw-r--r--  1 yarn hadoop    283072 Feb 23  2018 rm-audit.log.2018-02-23
-rw-r--r--  1 yarn hadoop    428690 Feb 24  2018 rm-audit.log.2018-02-24
-rw-r--r--  1 yarn hadoop    531180 Feb 25  2018 rm-audit.log.2018-02-25
-rw-r--r--  1 yarn hadoop    789163 Feb 26 23:23 rm-audit.log.2018-02-26
-rw-r--r--  1 yarn hadoop    682866 Feb 27 23:04 rm-audit.log.2018-02-27
-rw-r--r--  1 yarn hadoop    215674 Feb 28 23:02 rm-audit.log.2018-02-28
-rw-r--r--  1 yarn hadoop    401116 Mar  1 23:02 rm-audit.log.2018-03-01
-rw-r--r--  1 yarn hadoop    213927 Mar  2 23:02 rm-audit.log.2018-03-02
-rw-r--r--  1 yarn hadoop    213496 Mar  3 23:02 rm-audit.log.2018-03-03
-rw-r--r--  1 yarn hadoop    507271 Mar  4 23:01 rm-audit.log.2018-03-04
-rw-r--r--  1 yarn hadoop    210537 Mar  5 23:01 rm-audit.log.2018-03-05
-rw-r--r--  1 yarn hadoop   1604216 Mar  6 23:19 rm-audit.log.2018-03-06
-rw-r--r--  1 yarn hadoop    754952 Mar  7 23:19 rm-audit.log.2018-03-07
-rw-r--r--  1 yarn hadoop    745380 Mar  8 23:19 rm-audit.log.2018-03-08
-rw-r--r--  1 yarn hadoop    652800 Mar  9 23:19 rm-audit.log.2018-03-09
-rw-r--r--  1 yarn hadoop    653214 Mar 10 23:18 rm-audit.log.2018-03-10
-rw-r--r--  1 yarn hadoop   1342391 Mar 11 23:19 rm-audit.log.2018-03-11
-rw-r--r--  1 yarn hadoop    844104 Mar 12 23:19 rm-audit.log.2018-03-12
-rw-r--r--  1 yarn hadoop    473348 Mar 13 15:56 rm-audit.log.2018-03-13
-rw-r--r--  1 yarn hadoop       548 Mar 14 09:39 rm-audit.log.2018-03-14
-rw-r--r--  1 yarn hadoop       274 Mar 19 14:08 rm-audit.log.2018-03-19
-rw-r--r--  1 yarn hadoop       548 Apr 24 22:00 rm-audit.log.2018-04-24
-rw-r--r--  1 yarn hadoop      2192 Apr 25 23:08 rm-audit.log.2018-04-25
-rw-r--r--  1 yarn hadoop      3562 Apr 26 08:59 rm-audit.log.2018-04-26
-rw-r--r--  1 yarn hadoop       274 Apr 30 11:12 rm-audit.log.2018-04-30
-rw-r--r--  1 yarn hadoop       274 May  1 11:13 rm-audit.log.2018-05-01
-rw-r--r--  1 yarn hadoop       274 May  2 08:16 rm-audit.log.2018-05-02
-rw-r--r--  1 yarn hadoop       191 May  7 06:25 rm-audit.log.2018-05-07
-rw-r--r--  1 yarn hadoop       382 May 22 05:16 rm-audit.log.2018-05-22
-rw-r--r--  1 yarn hadoop      1096 Jun  6 06:06 rm-audit.log.2018-06-06
-rw-r--r--  1 yarn hadoop       548 Jun 14 12:02 rm-audit.log.2018-06-14
-rw-r--r--  1 yarn hadoop       274 Jun 24 12:33 rm-audit.log.2018-06-24
-rw-r--r--  1 yarn hadoop       274 Jun 25 07:12 rm-audit.log.2018-06-25
-rw-r--r--  1 yarn hadoop       274 Jun 27 18:01 rm-audit.log.2018-06-27
-rw-r--r--  1 yarn hadoop  10999618 Jun 28 14:25 rm-audit.log.2018-06-28
-rw-r--r--  1 yarn hadoop  11012326 Jul  1 12:25 rm-audit.log.2018-07-01
-rw-r--r--  1 yarn hadoop    114680 Jul  3 14:30 rm-audit.log.2018-07-03
-rw-r--r--  1 yarn hadoop  11004749 Jul  4 11:42 rm-audit.log.2018-07-04
-rw-r--r--  1 yarn hadoop     20055 Jul  5 12:20 rm-audit.log.2018-07-05
-rw-r--r--  1 yarn hadoop       132 Jul  7 19:10 rm-audit.log.2018-07-07
-rw-r--r--  1 yarn hadoop       274 Jul 16 09:02 rm-audit.log.2018-07-16
-rw-r--r--  1 yarn hadoop  18432816 Jul 17 23:59 rm-audit.log.2018-07-17

Re: how to configure the log4j in order to rotate the rm-audit files

apappu — Sun, 18 Aug 2019 00:31:21 GMT

@Michael Bronson

Did you tried modifying below configs in Ambari?

yarn_rm_summary_log_max_backup_size

yarn_rm_summary_log_number_of_backup_files

Re: how to configure the log4j in order to rotate the rm-audit files

mike_bronson7 — Mon, 27 Aug 2018 23:46:02 GMT

yes these values already configured

Re: how to configure the log4j in order to rotate the rm-audit files

mike_bronson7 — Mon, 27 Aug 2018 23:47:40 GMT

I succeeded with the following conf


# Audit logging for NodeManager
nm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false
#log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log
log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.NMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.NMAUDIT.MaxBackupIndex=5
log4j.appender.NMAUDIT.MaxFileSize=100KB

Re: how to configure the log4j in order to rotate the rm-audit files

mike_bronson7 — Mon, 27 Aug 2018 23:50:07 GMT

can you help me with this quastion - https://community.hortonworks.com/questions/214906/advanced-yarn-log4j-not-fit-what-we-get-in-varlogh.html

Re: how to configure the log4j in order to rotate the rm-audit files

apappu — Tue, 28 Aug 2018 00:10:00 GMT

@Michael Bronson

Below config would work fine. not sure DailyRollingFileAppender would work fine in case of size based rolling.

# Audit logging for ResourceManager
rm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.RMAUDIT.MaxFileSize=1KB
log4j.appender.RMAUDIT.MaxBackupIndex=2

Re: how to configure the log4j in order to rotate the rm-audit files

apappu — Tue, 28 Aug 2018 23:11:07 GMT

@Michael Bronson can you please select correct answer and end this thread?

question Re: how to configure the log4j in order to rotate the rm-audit files in Support Questions

how to configure the log4j in order to rotate the rm-audit files

Re: how to configure the log4j in order to rotate the rm-audit files

Re: how to configure the log4j in order to rotate the rm-audit files

Re: how to configure the log4j in order to rotate the rm-audit files

Re: how to configure the log4j in order to rotate the rm-audit files

Re: how to configure the log4j in order to rotate the rm-audit files

Re: how to configure the log4j in order to rotate the rm-audit files