Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to configure the log4j in order to rotate the rm-audit files

avatar

hi all,

Under /var/log/hadoop-yarn/yarn , we have the following files ( pic 1) :

how to reconfigure the Advanced yarn-log4j ( under YARN --> CONFIG ) , in order to rotate the files

so max backup files will be - 10 , and max file size will limit to 10K

yarn-log4j.txt

PIC 1

ls -ltr
total 2023884
-rw-r--r--. 1 yarn hadoop       274 Aug  7  2017 rm-audit.log.2017-08-07
-rw-r--r--. 1 yarn hadoop      1644 Aug  9  2017 rm-audit.log.2017-08-09
-rw-r--r--. 1 yarn hadoop      4346 Aug 10  2017 rm-audit.log.2017-08-10
-rw-r--r--. 1 yarn hadoop      6450 Aug 15  2017 rm-audit.log.2017-08-15
-rw-r--r--. 1 yarn hadoop       824 Aug 16  2017 rm-audit.log.2017-08-16
-rw-r--r--. 1 yarn hadoop     65573 Aug 17  2017 rm-audit.log.2017-08-17
-rw-r--r--. 1 yarn hadoop       274 Sep  6  2017 rm-audit.log.2017-09-06
-rw-r--r--. 1 yarn hadoop       274 Sep 24  2017 rm-audit.log.2017-09-24
-rw-r--r--  1 yarn hadoop       822 Sep 27  2017 rm-audit.log.2017-09-27
-rw-r--r--  1 yarn hadoop    135508 Sep 28  2017 rm-audit.log.2017-09-28
-rw-r--r--  1 yarn hadoop      3838 Oct  1  2017 rm-audit.log.2017-10-01
-rw-r--r--  1 yarn hadoop    175181 Oct  2  2017 rm-audit.log.2017-10-02
-rw-r--r--  1 yarn hadoop     54213 Oct  3  2017 rm-audit.log.2017-10-03
-rw-r--r--  1 yarn hadoop      5313 Oct  8  2017 rm-audit.log.2017-10-08
-rw-r--r--  1 yarn hadoop     32125 Oct  9  2017 rm-audit.log.2017-10-09
-rw-r--r--  1 yarn hadoop     25134 Oct 10  2017 rm-audit.log.2017-10-10
-rw-r--r--  1 yarn hadoop     89328 Oct 11  2017 rm-audit.log.2017-10-11
-rw-r--r--  1 yarn hadoop     47737 Oct 15  2017 rm-audit.log.2017-10-15
-rw-r--r--  1 yarn hadoop     14218 Oct 16  2017 rm-audit.log.2017-10-16
-rw-r--r--  1 yarn hadoop    126010 Oct 17  2017 rm-audit.log.2017-10-17
-rw-r--r--  1 yarn hadoop     89089 Oct 18  2017 rm-audit.log.2017-10-18
-rw-r--r--  1 yarn hadoop    157802 Oct 19  2017 rm-audit.log.2017-10-19
-rw-r--r--  1 yarn hadoop     23173 Oct 20  2017 rm-audit.log.2017-10-20
-rw-r--r--  1 yarn hadoop     25958 Oct 21  2017 rm-audit.log.2017-10-21
-rw-r--r--  1 yarn hadoop     22483 Oct 22  2017 rm-audit.log.2017-10-22
-rw-r--r--  1 yarn hadoop    153963 Oct 23  2017 rm-audit.log.2017-10-23
-rw-r--r--  1 yarn hadoop     82562 Oct 24  2017 rm-audit.log.2017-10-24
-rw-r--r--  1 yarn hadoop       274 Oct 29  2017 rm-audit.log.2017-10-29
-rw-r--r--  1 yarn hadoop   2139002 Oct 30  2017 rm-audit.log.2017-10-30
-rw-r--r--  1 yarn hadoop    969480 Oct 31  2017 rm-audit.log.2017-10-31
-rw-r--r--  1 yarn hadoop    991997 Nov  1  2017 rm-audit.log.2017-11-01
-rw-r--r--  1 yarn hadoop    994624 Nov  2  2017 rm-audit.log.2017-11-02
-rw-r--r--  1 yarn hadoop    978260 Nov  3  2017 rm-audit.log.2017-11-03
-rw-r--r--  1 yarn hadoop    977769 Nov  4  2017 rm-audit.log.2017-11-04
-rw-r--r--  1 yarn hadoop   1425299 Nov  5  2017 rm-audit.log.2017-11-05
-rw-r--r--  1 yarn hadoop   1499223 Nov  6  2017 rm-audit.log.2017-11-06
-rw-r--r--  1 yarn hadoop   1535388 Nov  7  2017 rm-audit.log.2017-11-07
-rw-r--r--  1 yarn hadoop       274 Nov 21  2017 rm-audit.log.2017-11-21
-rw-r--r--  1 yarn hadoop   4197197 Nov 27  2017 rm-audit.log.2017-11-27
-rw-r--r--  1 yarn hadoop   1145099 Nov 28  2017 rm-audit.log.2017-11-28
-rw-r--r--  1 yarn hadoop    802082 Nov 29  2017 rm-audit.log.2017-11-29
-rw-r--r--  1 yarn hadoop    623300 Nov 30  2017 rm-audit.log.2017-11-30
-rw-r--r--  1 yarn hadoop    636433 Dec  1  2017 rm-audit.log.2017-12-01
-rw-r--r--  1 yarn hadoop    798816 Dec  2  2017 rm-audit.log.2017-12-02
-rw-r--r--  1 yarn hadoop   1040673 Dec  3  2017 rm-audit.log.2017-12-03
-rw-r--r--  1 yarn hadoop    835143 Dec  4  2017 rm-audit.log.2017-12-04
-rw-r--r--  1 yarn hadoop   4000819 Dec  5  2017 rm-audit.log.2017-12-05
-rw-r--r--  1 yarn hadoop  33406088 Dec  6  2017 rm-audit.log.2017-12-06
-rw-r--r--  1 yarn hadoop  21565330 Dec  7  2017 rm-audit.log.2017-12-07
-rw-r--r--  1 yarn hadoop       274 Dec 11  2017 rm-audit.log.2017-12-11
-rw-r--r--  1 yarn hadoop       191 Dec 12  2017 rm-audit.log.2017-12-12
-rw-r--r--  1 yarn hadoop       274 Dec 14  2017 rm-audit.log.2017-12-14
-rw-r--r--  1 yarn hadoop       548 Dec 17  2017 rm-audit.log.2017-12-17
-rw-r--r--  1 yarn hadoop      2192 Dec 19  2017 rm-audit.log.2017-12-19
-rw-r--r--  1 yarn hadoop       274 Dec 27  2017 rm-audit.log.2017-12-27
-rw-r--r--  1 yarn hadoop   2754580 Dec 31  2017 rm-audit.log.2017-12-31
-rw-r--r--  1 yarn hadoop   1951645 Jan  1  2018 rm-audit.log.2018-01-01
-rw-r--r--  1 yarn hadoop       548 Jan  3  2018 rm-audit.log.2018-01-03
-rw-r--r--  1 yarn hadoop       274 Jan  8  2018 rm-audit.log.2018-01-08
-rw-r--r--  1 yarn hadoop       274 Jan 10  2018 rm-audit.log.2018-01-10
-rw-r--r--  1 yarn hadoop   2145304 Jan 11  2018 rm-audit.log.2018-01-11
-rw-r--r--  1 yarn hadoop    197143 Jan 13  2018 rm-audit.log.2018-01-13
-rw-r--r--  1 yarn hadoop      1370 Jan 14  2018 rm-audit.log.2018-01-14
-rw-r--r--  1 yarn hadoop       274 Jan 15  2018 rm-audit.log.2018-01-15
-rw-r--r--  1 yarn hadoop       274 Jan 18  2018 rm-audit.log.2018-01-18
-rw-r--r--  1 yarn hadoop   2363585 Jan 19  2018 rm-audit.log.2018-01-19
-rw-r--r--  1 yarn hadoop       822 Jan 20  2018 rm-audit.log.2018-01-20
-rw-r--r--  1 yarn hadoop   7395951 Jan 21  2018 rm-audit.log.2018-01-21
-rw-r--r--  1 yarn hadoop    241166 Jan 22  2018 rm-audit.log.2018-01-22
-rw-r--r--  1 yarn hadoop     19976 Jan 23  2018 rm-audit.log.2018-01-23
-rw-r--r--  1 yarn hadoop  10999019 Jan 24  2018 rm-audit.log.2018-01-24
-rw-r--r--  1 yarn hadoop  17312694 Jan 25  2018 rm-audit.log.2018-01-25
-rw-r--r--  1 yarn hadoop  31321833 Jan 26  2018 rm-audit.log.2018-01-26
-rw-r--r--  1 yarn hadoop  21793442 Jan 27  2018 rm-audit.log.2018-01-27
-rw-r--r--  1 yarn hadoop  28805081 Jan 28  2018 rm-audit.log.2018-01-28
-rw-r--r--  1 yarn hadoop  32190177 Jan 29  2018 rm-audit.log.2018-01-29
-rw-r--r--  1 yarn hadoop  21629686 Jan 30  2018 rm-audit.log.2018-01-30
-rw-r--r--  1 yarn hadoop       274 Feb  1  2018 rm-audit.log.2018-02-01
-rw-r--r--  1 yarn hadoop  15550664 Feb  4  2018 rm-audit.log.2018-02-04
-rw-r--r--  1 yarn hadoop       548 Feb  6  2018 rm-audit.log.2018-02-06
-rw-r--r--  1 yarn hadoop  12077408 Feb 13  2018 rm-audit.log.2018-02-13
-rw-r--r--  1 yarn hadoop    403786 Feb 14  2018 rm-audit.log.2018-02-14
-rw-r--r--  1 yarn hadoop    421911 Feb 15  2018 rm-audit.log.2018-02-15
-rw-r--r--  1 yarn hadoop    202958 Feb 16  2018 rm-audit.log.2018-02-16
-rw-r--r--  1 yarn hadoop    208080 Feb 17  2018 rm-audit.log.2018-02-17
-rw-r--r--  1 yarn hadoop    485789 Feb 18  2018 rm-audit.log.2018-02-18
-rw-r--r--  1 yarn hadoop    753580 Feb 19  2018 rm-audit.log.2018-02-19
-rw-r--r--  1 yarn hadoop    389283 Feb 20  2018 rm-audit.log.2018-02-20
-rw-r--r--  1 yarn hadoop    403405 Feb 21  2018 rm-audit.log.2018-02-21
-rw-r--r--  1 yarn hadoop    475882 Feb 22  2018 rm-audit.log.2018-02-22
-rw-r--r--  1 yarn hadoop    283072 Feb 23  2018 rm-audit.log.2018-02-23
-rw-r--r--  1 yarn hadoop    428690 Feb 24  2018 rm-audit.log.2018-02-24
-rw-r--r--  1 yarn hadoop    531180 Feb 25  2018 rm-audit.log.2018-02-25
-rw-r--r--  1 yarn hadoop    789163 Feb 26 23:23 rm-audit.log.2018-02-26
-rw-r--r--  1 yarn hadoop    682866 Feb 27 23:04 rm-audit.log.2018-02-27
-rw-r--r--  1 yarn hadoop    215674 Feb 28 23:02 rm-audit.log.2018-02-28
-rw-r--r--  1 yarn hadoop    401116 Mar  1 23:02 rm-audit.log.2018-03-01
-rw-r--r--  1 yarn hadoop    213927 Mar  2 23:02 rm-audit.log.2018-03-02
-rw-r--r--  1 yarn hadoop    213496 Mar  3 23:02 rm-audit.log.2018-03-03
-rw-r--r--  1 yarn hadoop    507271 Mar  4 23:01 rm-audit.log.2018-03-04
-rw-r--r--  1 yarn hadoop    210537 Mar  5 23:01 rm-audit.log.2018-03-05
-rw-r--r--  1 yarn hadoop   1604216 Mar  6 23:19 rm-audit.log.2018-03-06
-rw-r--r--  1 yarn hadoop    754952 Mar  7 23:19 rm-audit.log.2018-03-07
-rw-r--r--  1 yarn hadoop    745380 Mar  8 23:19 rm-audit.log.2018-03-08
-rw-r--r--  1 yarn hadoop    652800 Mar  9 23:19 rm-audit.log.2018-03-09
-rw-r--r--  1 yarn hadoop    653214 Mar 10 23:18 rm-audit.log.2018-03-10
-rw-r--r--  1 yarn hadoop   1342391 Mar 11 23:19 rm-audit.log.2018-03-11
-rw-r--r--  1 yarn hadoop    844104 Mar 12 23:19 rm-audit.log.2018-03-12
-rw-r--r--  1 yarn hadoop    473348 Mar 13 15:56 rm-audit.log.2018-03-13
-rw-r--r--  1 yarn hadoop       548 Mar 14 09:39 rm-audit.log.2018-03-14
-rw-r--r--  1 yarn hadoop       274 Mar 19 14:08 rm-audit.log.2018-03-19
-rw-r--r--  1 yarn hadoop       548 Apr 24 22:00 rm-audit.log.2018-04-24
-rw-r--r--  1 yarn hadoop      2192 Apr 25 23:08 rm-audit.log.2018-04-25
-rw-r--r--  1 yarn hadoop      3562 Apr 26 08:59 rm-audit.log.2018-04-26
-rw-r--r--  1 yarn hadoop       274 Apr 30 11:12 rm-audit.log.2018-04-30
-rw-r--r--  1 yarn hadoop       274 May  1 11:13 rm-audit.log.2018-05-01
-rw-r--r--  1 yarn hadoop       274 May  2 08:16 rm-audit.log.2018-05-02
-rw-r--r--  1 yarn hadoop       191 May  7 06:25 rm-audit.log.2018-05-07
-rw-r--r--  1 yarn hadoop       382 May 22 05:16 rm-audit.log.2018-05-22
-rw-r--r--  1 yarn hadoop      1096 Jun  6 06:06 rm-audit.log.2018-06-06
-rw-r--r--  1 yarn hadoop       548 Jun 14 12:02 rm-audit.log.2018-06-14
-rw-r--r--  1 yarn hadoop       274 Jun 24 12:33 rm-audit.log.2018-06-24
-rw-r--r--  1 yarn hadoop       274 Jun 25 07:12 rm-audit.log.2018-06-25
-rw-r--r--  1 yarn hadoop       274 Jun 27 18:01 rm-audit.log.2018-06-27
-rw-r--r--  1 yarn hadoop  10999618 Jun 28 14:25 rm-audit.log.2018-06-28
-rw-r--r--  1 yarn hadoop  11012326 Jul  1 12:25 rm-audit.log.2018-07-01
-rw-r--r--  1 yarn hadoop    114680 Jul  3 14:30 rm-audit.log.2018-07-03
-rw-r--r--  1 yarn hadoop  11004749 Jul  4 11:42 rm-audit.log.2018-07-04
-rw-r--r--  1 yarn hadoop     20055 Jul  5 12:20 rm-audit.log.2018-07-05
-rw-r--r--  1 yarn hadoop       132 Jul  7 19:10 rm-audit.log.2018-07-07
-rw-r--r--  1 yarn hadoop       274 Jul 16 09:02 rm-audit.log.2018-07-16
-rw-r--r--  1 yarn hadoop  18432816 Jul 17 23:59 rm-audit.log.2018-07-17
Michael-Bronson
1 ACCEPTED SOLUTION

avatar

@Michael Bronson

Did you tried modifying below configs in Ambari?

yarn_rm_summary_log_max_backup_size

yarn_rm_summary_log_number_of_backup_files

85757-screen-shot-2018-08-27-at-93100-am.png

View solution in original post

6 REPLIES 6

avatar

@Michael Bronson

Did you tried modifying below configs in Ambari?

yarn_rm_summary_log_max_backup_size

yarn_rm_summary_log_number_of_backup_files

85757-screen-shot-2018-08-27-at-93100-am.png

avatar

yes these values already configured

Michael-Bronson

avatar

I succeeded with the following conf


# Audit logging for NodeManager nm.audit.logger=${hadoop.root.logger} log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger} log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false #log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n #log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.NMAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.NMAUDIT.MaxBackupIndex=5 log4j.appender.NMAUDIT.MaxFileSize=100KB
Michael-Bronson

avatar

@Michael Bronson

Below config would work fine. not sure DailyRollingFileAppender would work fine in case of size based rolling.

# Audit logging for ResourceManager
rm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.RMAUDIT.MaxFileSize=1KB
log4j.appender.RMAUDIT.MaxBackupIndex=2

avatar

@Michael Bronson can you please select correct answer and end this thread?

avatar