we manage many hadoop clusters based on redhat OS ( version 7.x ) based on our experience ( many problem of low memory , disks performance , network problem , etc ) we agree that we need to install some monitoring tool that have ability to save the monitoring at least one month history details from the link below https://neverendingsecurity.wordpress.com/tag/atop/ we saw a lot of monitoring tool and we not sure what is the best tool for hadoop clusters , meanwhile we install the atop tool that its fine ( but take a lot of space under /var/log/atop ) **but we still thinking if this is good selecting** ... View more

hi all, we configured the Audit logging for ResourceManager , ( in YARN --> Advanced yarn-log4j ) as the following: # Audit logging for ResourceManager rm.audit.logger=${hadoop.root.logger} log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false #log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n #log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.RMAUDIT.MaxBackupIndex=5 log4j.appender.RMAUDIT.MaxFileSize=1MB and # Audit logging for NodeManager nm.audit.logger=${hadoop.root.logger} log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger} log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false #log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n #log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.NMAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.NMAUDIT.MaxBackupIndex=5 log4j.appender.NMAUDIT.MaxFileSize=1MB and restart the YARN service now we want to see if rm-audit.log and nm-audit.log are rotated , but because we set 1MB in MaxFileSize it will take time until logs will be rotated -rw-r--r-- 1 yarn hadoop 4180 Aug 28 09:32 rm-audit.log is it possible to in force the logs to increase quickly to 1M and rotated as rm-audit.log.1 , and rm-audit.log.2 , etc so we can sure that our log4j configuration is correctly ? ... View more

hi all, Under /var/log/hadoop-yarn/yarn , we have the following files ( pic 1) : how to reconfigure the Advanced yarn-log4j ( under YARN --> CONFIG ) , in order to rotate the files so max backup files will be - 10 , and max file size will limit to 10K yarn-log4j.txt PIC 1 ls -ltr total 2023884 -rw-r--r--. 1 yarn hadoop 274 Aug 7 2017 rm-audit.log.2017-08-07 -rw-r--r--. 1 yarn hadoop 1644 Aug 9 2017 rm-audit.log.2017-08-09 -rw-r--r--. 1 yarn hadoop 4346 Aug 10 2017 rm-audit.log.2017-08-10 -rw-r--r--. 1 yarn hadoop 6450 Aug 15 2017 rm-audit.log.2017-08-15 -rw-r--r--. 1 yarn hadoop 824 Aug 16 2017 rm-audit.log.2017-08-16 -rw-r--r--. 1 yarn hadoop 65573 Aug 17 2017 rm-audit.log.2017-08-17 -rw-r--r--. 1 yarn hadoop 274 Sep 6 2017 rm-audit.log.2017-09-06 -rw-r--r--. 1 yarn hadoop 274 Sep 24 2017 rm-audit.log.2017-09-24 -rw-r--r-- 1 yarn hadoop 822 Sep 27 2017 rm-audit.log.2017-09-27 -rw-r--r-- 1 yarn hadoop 135508 Sep 28 2017 rm-audit.log.2017-09-28 -rw-r--r-- 1 yarn hadoop 3838 Oct 1 2017 rm-audit.log.2017-10-01 -rw-r--r-- 1 yarn hadoop 175181 Oct 2 2017 rm-audit.log.2017-10-02 -rw-r--r-- 1 yarn hadoop 54213 Oct 3 2017 rm-audit.log.2017-10-03 -rw-r--r-- 1 yarn hadoop 5313 Oct 8 2017 rm-audit.log.2017-10-08 -rw-r--r-- 1 yarn hadoop 32125 Oct 9 2017 rm-audit.log.2017-10-09 -rw-r--r-- 1 yarn hadoop 25134 Oct 10 2017 rm-audit.log.2017-10-10 -rw-r--r-- 1 yarn hadoop 89328 Oct 11 2017 rm-audit.log.2017-10-11 -rw-r--r-- 1 yarn hadoop 47737 Oct 15 2017 rm-audit.log.2017-10-15 -rw-r--r-- 1 yarn hadoop 14218 Oct 16 2017 rm-audit.log.2017-10-16 -rw-r--r-- 1 yarn hadoop 126010 Oct 17 2017 rm-audit.log.2017-10-17 -rw-r--r-- 1 yarn hadoop 89089 Oct 18 2017 rm-audit.log.2017-10-18 -rw-r--r-- 1 yarn hadoop 157802 Oct 19 2017 rm-audit.log.2017-10-19 -rw-r--r-- 1 yarn hadoop 23173 Oct 20 2017 rm-audit.log.2017-10-20 -rw-r--r-- 1 yarn hadoop 25958 Oct 21 2017 rm-audit.log.2017-10-21 -rw-r--r-- 1 yarn hadoop 22483 Oct 22 2017 rm-audit.log.2017-10-22 -rw-r--r-- 1 yarn hadoop 153963 Oct 23 2017 rm-audit.log.2017-10-23 -rw-r--r-- 1 yarn hadoop 82562 Oct 24 2017 rm-audit.log.2017-10-24 -rw-r--r-- 1 yarn hadoop 274 Oct 29 2017 rm-audit.log.2017-10-29 -rw-r--r-- 1 yarn hadoop 2139002 Oct 30 2017 rm-audit.log.2017-10-30 -rw-r--r-- 1 yarn hadoop 969480 Oct 31 2017 rm-audit.log.2017-10-31 -rw-r--r-- 1 yarn hadoop 991997 Nov 1 2017 rm-audit.log.2017-11-01 -rw-r--r-- 1 yarn hadoop 994624 Nov 2 2017 rm-audit.log.2017-11-02 -rw-r--r-- 1 yarn hadoop 978260 Nov 3 2017 rm-audit.log.2017-11-03 -rw-r--r-- 1 yarn hadoop 977769 Nov 4 2017 rm-audit.log.2017-11-04 -rw-r--r-- 1 yarn hadoop 1425299 Nov 5 2017 rm-audit.log.2017-11-05 -rw-r--r-- 1 yarn hadoop 1499223 Nov 6 2017 rm-audit.log.2017-11-06 -rw-r--r-- 1 yarn hadoop 1535388 Nov 7 2017 rm-audit.log.2017-11-07 -rw-r--r-- 1 yarn hadoop 274 Nov 21 2017 rm-audit.log.2017-11-21 -rw-r--r-- 1 yarn hadoop 4197197 Nov 27 2017 rm-audit.log.2017-11-27 -rw-r--r-- 1 yarn hadoop 1145099 Nov 28 2017 rm-audit.log.2017-11-28 -rw-r--r-- 1 yarn hadoop 802082 Nov 29 2017 rm-audit.log.2017-11-29 -rw-r--r-- 1 yarn hadoop 623300 Nov 30 2017 rm-audit.log.2017-11-30 -rw-r--r-- 1 yarn hadoop 636433 Dec 1 2017 rm-audit.log.2017-12-01 -rw-r--r-- 1 yarn hadoop 798816 Dec 2 2017 rm-audit.log.2017-12-02 -rw-r--r-- 1 yarn hadoop 1040673 Dec 3 2017 rm-audit.log.2017-12-03 -rw-r--r-- 1 yarn hadoop 835143 Dec 4 2017 rm-audit.log.2017-12-04 -rw-r--r-- 1 yarn hadoop 4000819 Dec 5 2017 rm-audit.log.2017-12-05 -rw-r--r-- 1 yarn hadoop 33406088 Dec 6 2017 rm-audit.log.2017-12-06 -rw-r--r-- 1 yarn hadoop 21565330 Dec 7 2017 rm-audit.log.2017-12-07 -rw-r--r-- 1 yarn hadoop 274 Dec 11 2017 rm-audit.log.2017-12-11 -rw-r--r-- 1 yarn hadoop 191 Dec 12 2017 rm-audit.log.2017-12-12 -rw-r--r-- 1 yarn hadoop 274 Dec 14 2017 rm-audit.log.2017-12-14 -rw-r--r-- 1 yarn hadoop 548 Dec 17 2017 rm-audit.log.2017-12-17 -rw-r--r-- 1 yarn hadoop 2192 Dec 19 2017 rm-audit.log.2017-12-19 -rw-r--r-- 1 yarn hadoop 274 Dec 27 2017 rm-audit.log.2017-12-27 -rw-r--r-- 1 yarn hadoop 2754580 Dec 31 2017 rm-audit.log.2017-12-31 -rw-r--r-- 1 yarn hadoop 1951645 Jan 1 2018 rm-audit.log.2018-01-01 -rw-r--r-- 1 yarn hadoop 548 Jan 3 2018 rm-audit.log.2018-01-03 -rw-r--r-- 1 yarn hadoop 274 Jan 8 2018 rm-audit.log.2018-01-08 -rw-r--r-- 1 yarn hadoop 274 Jan 10 2018 rm-audit.log.2018-01-10 -rw-r--r-- 1 yarn hadoop 2145304 Jan 11 2018 rm-audit.log.2018-01-11 -rw-r--r-- 1 yarn hadoop 197143 Jan 13 2018 rm-audit.log.2018-01-13 -rw-r--r-- 1 yarn hadoop 1370 Jan 14 2018 rm-audit.log.2018-01-14 -rw-r--r-- 1 yarn hadoop 274 Jan 15 2018 rm-audit.log.2018-01-15 -rw-r--r-- 1 yarn hadoop 274 Jan 18 2018 rm-audit.log.2018-01-18 -rw-r--r-- 1 yarn hadoop 2363585 Jan 19 2018 rm-audit.log.2018-01-19 -rw-r--r-- 1 yarn hadoop 822 Jan 20 2018 rm-audit.log.2018-01-20 -rw-r--r-- 1 yarn hadoop 7395951 Jan 21 2018 rm-audit.log.2018-01-21 -rw-r--r-- 1 yarn hadoop 241166 Jan 22 2018 rm-audit.log.2018-01-22 -rw-r--r-- 1 yarn hadoop 19976 Jan 23 2018 rm-audit.log.2018-01-23 -rw-r--r-- 1 yarn hadoop 10999019 Jan 24 2018 rm-audit.log.2018-01-24 -rw-r--r-- 1 yarn hadoop 17312694 Jan 25 2018 rm-audit.log.2018-01-25 -rw-r--r-- 1 yarn hadoop 31321833 Jan 26 2018 rm-audit.log.2018-01-26 -rw-r--r-- 1 yarn hadoop 21793442 Jan 27 2018 rm-audit.log.2018-01-27 -rw-r--r-- 1 yarn hadoop 28805081 Jan 28 2018 rm-audit.log.2018-01-28 -rw-r--r-- 1 yarn hadoop 32190177 Jan 29 2018 rm-audit.log.2018-01-29 -rw-r--r-- 1 yarn hadoop 21629686 Jan 30 2018 rm-audit.log.2018-01-30 -rw-r--r-- 1 yarn hadoop 274 Feb 1 2018 rm-audit.log.2018-02-01 -rw-r--r-- 1 yarn hadoop 15550664 Feb 4 2018 rm-audit.log.2018-02-04 -rw-r--r-- 1 yarn hadoop 548 Feb 6 2018 rm-audit.log.2018-02-06 -rw-r--r-- 1 yarn hadoop 12077408 Feb 13 2018 rm-audit.log.2018-02-13 -rw-r--r-- 1 yarn hadoop 403786 Feb 14 2018 rm-audit.log.2018-02-14 -rw-r--r-- 1 yarn hadoop 421911 Feb 15 2018 rm-audit.log.2018-02-15 -rw-r--r-- 1 yarn hadoop 202958 Feb 16 2018 rm-audit.log.2018-02-16 -rw-r--r-- 1 yarn hadoop 208080 Feb 17 2018 rm-audit.log.2018-02-17 -rw-r--r-- 1 yarn hadoop 485789 Feb 18 2018 rm-audit.log.2018-02-18 -rw-r--r-- 1 yarn hadoop 753580 Feb 19 2018 rm-audit.log.2018-02-19 -rw-r--r-- 1 yarn hadoop 389283 Feb 20 2018 rm-audit.log.2018-02-20 -rw-r--r-- 1 yarn hadoop 403405 Feb 21 2018 rm-audit.log.2018-02-21 -rw-r--r-- 1 yarn hadoop 475882 Feb 22 2018 rm-audit.log.2018-02-22 -rw-r--r-- 1 yarn hadoop 283072 Feb 23 2018 rm-audit.log.2018-02-23 -rw-r--r-- 1 yarn hadoop 428690 Feb 24 2018 rm-audit.log.2018-02-24 -rw-r--r-- 1 yarn hadoop 531180 Feb 25 2018 rm-audit.log.2018-02-25 -rw-r--r-- 1 yarn hadoop 789163 Feb 26 23:23 rm-audit.log.2018-02-26 -rw-r--r-- 1 yarn hadoop 682866 Feb 27 23:04 rm-audit.log.2018-02-27 -rw-r--r-- 1 yarn hadoop 215674 Feb 28 23:02 rm-audit.log.2018-02-28 -rw-r--r-- 1 yarn hadoop 401116 Mar 1 23:02 rm-audit.log.2018-03-01 -rw-r--r-- 1 yarn hadoop 213927 Mar 2 23:02 rm-audit.log.2018-03-02 -rw-r--r-- 1 yarn hadoop 213496 Mar 3 23:02 rm-audit.log.2018-03-03 -rw-r--r-- 1 yarn hadoop 507271 Mar 4 23:01 rm-audit.log.2018-03-04 -rw-r--r-- 1 yarn hadoop 210537 Mar 5 23:01 rm-audit.log.2018-03-05 -rw-r--r-- 1 yarn hadoop 1604216 Mar 6 23:19 rm-audit.log.2018-03-06 -rw-r--r-- 1 yarn hadoop 754952 Mar 7 23:19 rm-audit.log.2018-03-07 -rw-r--r-- 1 yarn hadoop 745380 Mar 8 23:19 rm-audit.log.2018-03-08 -rw-r--r-- 1 yarn hadoop 652800 Mar 9 23:19 rm-audit.log.2018-03-09 -rw-r--r-- 1 yarn hadoop 653214 Mar 10 23:18 rm-audit.log.2018-03-10 -rw-r--r-- 1 yarn hadoop 1342391 Mar 11 23:19 rm-audit.log.2018-03-11 -rw-r--r-- 1 yarn hadoop 844104 Mar 12 23:19 rm-audit.log.2018-03-12 -rw-r--r-- 1 yarn hadoop 473348 Mar 13 15:56 rm-audit.log.2018-03-13 -rw-r--r-- 1 yarn hadoop 548 Mar 14 09:39 rm-audit.log.2018-03-14 -rw-r--r-- 1 yarn hadoop 274 Mar 19 14:08 rm-audit.log.2018-03-19 -rw-r--r-- 1 yarn hadoop 548 Apr 24 22:00 rm-audit.log.2018-04-24 -rw-r--r-- 1 yarn hadoop 2192 Apr 25 23:08 rm-audit.log.2018-04-25 -rw-r--r-- 1 yarn hadoop 3562 Apr 26 08:59 rm-audit.log.2018-04-26 -rw-r--r-- 1 yarn hadoop 274 Apr 30 11:12 rm-audit.log.2018-04-30 -rw-r--r-- 1 yarn hadoop 274 May 1 11:13 rm-audit.log.2018-05-01 -rw-r--r-- 1 yarn hadoop 274 May 2 08:16 rm-audit.log.2018-05-02 -rw-r--r-- 1 yarn hadoop 191 May 7 06:25 rm-audit.log.2018-05-07 -rw-r--r-- 1 yarn hadoop 382 May 22 05:16 rm-audit.log.2018-05-22 -rw-r--r-- 1 yarn hadoop 1096 Jun 6 06:06 rm-audit.log.2018-06-06 -rw-r--r-- 1 yarn hadoop 548 Jun 14 12:02 rm-audit.log.2018-06-14 -rw-r--r-- 1 yarn hadoop 274 Jun 24 12:33 rm-audit.log.2018-06-24 -rw-r--r-- 1 yarn hadoop 274 Jun 25 07:12 rm-audit.log.2018-06-25 -rw-r--r-- 1 yarn hadoop 274 Jun 27 18:01 rm-audit.log.2018-06-27 -rw-r--r-- 1 yarn hadoop 10999618 Jun 28 14:25 rm-audit.log.2018-06-28 -rw-r--r-- 1 yarn hadoop 11012326 Jul 1 12:25 rm-audit.log.2018-07-01 -rw-r--r-- 1 yarn hadoop 114680 Jul 3 14:30 rm-audit.log.2018-07-03 -rw-r--r-- 1 yarn hadoop 11004749 Jul 4 11:42 rm-audit.log.2018-07-04 -rw-r--r-- 1 yarn hadoop 20055 Jul 5 12:20 rm-audit.log.2018-07-05 -rw-r--r-- 1 yarn hadoop 132 Jul 7 19:10 rm-audit.log.2018-07-07 -rw-r--r-- 1 yarn hadoop 274 Jul 16 09:02 rm-audit.log.2018-07-16 -rw-r--r-- 1 yarn hadoop 18432816 Jul 17 23:59 rm-audit.log.2018-07-17 ... View more

hi all under /var/log/hadoop-yarn/yarn we have the following logs <strong> du -sh * 180K hadoop-mapreduce.jobsummary.log 448K rm-audit.log 132K rm-audit.log.2018-08-01 8.0K rm-audit.log.2018-08-02 4.0K rm-audit.log.2018-08-03 4.0K rm-audit.log.2018-08-06 8.0K rm-audit.log.2018-08-09 4.0K rm-audit.log.2018-08-12 8.0K rm-audit.log.2018-08-13 4.0K rm-audit.log.2018-08-17 36K rm-audit.log.2018-08-18 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.1 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.10 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.2 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.3 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.4 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.5 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.6 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.7 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.8 4.0K yarn-yarn-resourcemanager-master02.sys76.com.log.9 4.0K yarn-yarn-resourcemanager-master02.sys76.com.out 4.0K yarn-yarn-resourcemanager-master02.sys76.com.out.1 4.0K yarn-yarn-resourcemanager-master02.sys76.com.out.2 4.0K yarn-yarn-resourcemanager-master02.sys76.com.out.3 4.0K yarn-yarn-resourcemanager-master02.sys76.com.out.4 4.0K yarn-yarn-resourcemanager-master02.sys76.com.out.5</strong> <strong>[root@master02 yarn]# \ls -ltr total 904 -rw-r--r--. 1 yarn hadoop 2271 Aug 1 13:31 yarn-yarn-resourcemanager-master02.sys76.com.out.5 -rw-r--r--. 1 yarn hadoop 130412 Aug 1 23:58 rm-audit.log.2018-08-01 -rw-r--r-- 1 yarn hadoop 2052 Aug 2 01:53 yarn-yarn-resourcemanager-master02.sys76.com.out.4 -rw-r--r-- 1 yarn hadoop 1734 Aug 2 09:26 yarn-yarn-resourcemanager-master02.sys76.com.out.3 -rw-r--r-- 1 yarn hadoop 8186 Aug 2 23:26 rm-audit.log.2018-08-02 -rw-r--r-- 1 yarn hadoop 548 Aug 3 07:43 rm-audit.log.2018-08-03 -rw-r--r-- 1 yarn hadoop 274 Aug 6 09:20 rm-audit.log.2018-08-06 -rw-r--r-- 1 yarn hadoop 5480 Aug 9 15:36 rm-audit.log.2018-08-09 -rw-r--r-- 1 yarn hadoop 274 Aug 12 16:29 rm-audit.log.2018-08-12 -rw-r--r-- 1 yarn hadoop 6576 Aug 13 16:54 rm-audit.log.2018-08-13 -rw-r--r-- 1 yarn hadoop 1370 Aug 17 23:56 rm-audit.log.2018-08-17 -rw-r--r-- 1 yarn hadoop 1944 Aug 18 01:21 yarn-yarn-resourcemanager-master02.sys76.com.out.2 -rw-r--r-- 1 yarn hadoop 34937 Aug 18 03:46 rm-audit.log.2018-08-18 -rw-r--r--. 1 yarn hadoop 176470 Aug 27 12:03 hadoop-mapreduce.jobsummary.log -rw-r--r-- 1 yarn hadoop 1847 Aug 27 12:03 yarn-yarn-resourcemanager-master02.sys76.com.out.1 -rw-r--r-- 1 yarn hadoop 1052 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.10 -rw-r--r-- 1 yarn hadoop 1180 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.9 -rw-r--r-- 1 yarn hadoop 1126 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.8 -rw-r--r-- 1 yarn hadoop 1184 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.7 -rw-r--r-- 1 yarn hadoop 1184 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.6 -rw-r--r-- 1 yarn hadoop 1088 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.5 -rw-r--r-- 1 yarn hadoop 1158 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.4 -rw-r--r-- 1 yarn hadoop 1042 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.3 -rw-r--r-- 1 yarn hadoop 1147 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.2 -rw-r--r-- 1 yarn hadoop 1531 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.out -rw-r--r-- 1 yarn hadoop 1090 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.1 -rw-r--r-- 1 yarn hadoop 455502 Aug 27 12:06 rm-audit.log -rw-r--r-- 1 yarn hadoop 459 Aug 27 12:15 yarn-yarn-resourcemanager-master02.sys76.com.log [root@master02 yarn]# pwd /var/log/hadoop-yarn/yarn</strong> its very strange because according to Advanced yarn-log4j from YARN config we have diff conf we not understand why in /var/log/hadoop-yarn/yarn we get 10 backup files while yarn-log4j isn't define it second each yarn-yarn-resourcemanager-master02.sys76.com.log.XX , should get 256M as define in Advanced yarn-log4j , but from looking under /var/log/hadoop-yarn/yarn , each backup is only 4K so we not understand why each backup is 4K , and from where he get this value ? Advanced yarn-log4j ( from YARN CONFIG ) #Relative to Yarn Log Dir Prefix yarn.log.dir=. # # Job Summary Appender # # Use following logger to send summary to separate file defined by # hadoop.mapreduce.jobsummary.log.file rolled daily: # hadoop.mapreduce.jobsummary.logger=INFO,JSA # hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger} hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender # Set the ResourceManager summary log filename yarn.server.resourcemanager.appsummary.log.file=hadoop-mapreduce.jobsummary.log # Set the ResourceManager summary log level and appender yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger} #yarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY # To enable AppSummaryLogging for the RM, # set yarn.server.resourcemanager.appsummary.logger to # LEVEL,RMSUMMARY in hadoop-env.sh # Appender for ResourceManager Application Summary Log # Requires the following properties to be set # - hadoop.log.dir (Hadoop Log directory) # - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename) # - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender) log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender log4j.appender.RMSUMMARY.File=${yarn.log.dir}/${yarn.server.resourcemanager.appsummary.log.file} log4j.appender.RMSUMMARY.MaxFileSize=256MB log4j.appender.RMSUMMARY.MaxBackupIndex=20 log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.JSA.layout=org.apache.log4j.PatternLayout log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n log4j.appender.JSA.DatePattern=.yyyy-MM-dd log4j.appender.JSA.layout=org.apache.log4j.PatternLayout log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false # Appender for viewing information for errors and warnings yarn.ewma.cleanupInterval=300 yarn.ewma.messageAgeLimitSeconds=86400 yarn.ewma.maxUniqueMessages=250 log4j.appender.EWMA=org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender log4j.appender.EWMA.cleanupInterval=${yarn.ewma.cleanupInterval} log4j.appender.EWMA.messageAgeLimitSeconds=${yarn.ewma.messageAgeLimitSeconds} log4j.appender.EWMA.maxUniqueMessages=${yarn.ewma.maxUniqueMessages} # Audit logging for ResourceManager rm.audit.logger=${hadoop.root.logger} log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger} log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd # Audit logging for NodeManager nm.audit.logger=${hadoop.root.logger} log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger} log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd ... View more