Advanced yarn-log4j not fit what we get in /var/log/hadoop-yarn/yarn

[#1]

hi all

under /var/log/hadoop-yarn/yarn we have the following logs

<strong> du -sh *
180K    hadoop-mapreduce.jobsummary.log
448K    rm-audit.log
132K    rm-audit.log.2018-08-01
8.0K    rm-audit.log.2018-08-02
4.0K    rm-audit.log.2018-08-03
4.0K    rm-audit.log.2018-08-06
8.0K    rm-audit.log.2018-08-09
4.0K    rm-audit.log.2018-08-12
8.0K    rm-audit.log.2018-08-13
4.0K    rm-audit.log.2018-08-17
36K     rm-audit.log.2018-08-18
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.1
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.10
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.2
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.3
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.4
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.5
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.6
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.7
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.8
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.9
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.1
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.2
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.3
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.4
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.5</strong>

<strong>[root@master02 yarn]# \ls -ltr
total 904
-rw-r--r--. 1 yarn hadoop   2271 Aug  1 13:31 yarn-yarn-resourcemanager-master02.sys76.com.out.5
-rw-r--r--. 1 yarn hadoop 130412 Aug  1 23:58 rm-audit.log.2018-08-01
-rw-r--r--  1 yarn hadoop   2052 Aug  2 01:53 yarn-yarn-resourcemanager-master02.sys76.com.out.4
-rw-r--r--  1 yarn hadoop   1734 Aug  2 09:26 yarn-yarn-resourcemanager-master02.sys76.com.out.3
-rw-r--r--  1 yarn hadoop   8186 Aug  2 23:26 rm-audit.log.2018-08-02
-rw-r--r--  1 yarn hadoop    548 Aug  3 07:43 rm-audit.log.2018-08-03
-rw-r--r--  1 yarn hadoop    274 Aug  6 09:20 rm-audit.log.2018-08-06
-rw-r--r--  1 yarn hadoop   5480 Aug  9 15:36 rm-audit.log.2018-08-09
-rw-r--r--  1 yarn hadoop    274 Aug 12 16:29 rm-audit.log.2018-08-12
-rw-r--r--  1 yarn hadoop   6576 Aug 13 16:54 rm-audit.log.2018-08-13
-rw-r--r--  1 yarn hadoop   1370 Aug 17 23:56 rm-audit.log.2018-08-17
-rw-r--r--  1 yarn hadoop   1944 Aug 18 01:21 yarn-yarn-resourcemanager-master02.sys76.com.out.2
-rw-r--r--  1 yarn hadoop  34937 Aug 18 03:46 rm-audit.log.2018-08-18
-rw-r--r--. 1 yarn hadoop 176470 Aug 27 12:03 hadoop-mapreduce.jobsummary.log
-rw-r--r--  1 yarn hadoop   1847 Aug 27 12:03 yarn-yarn-resourcemanager-master02.sys76.com.out.1
-rw-r--r--  1 yarn hadoop   1052 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.10
-rw-r--r--  1 yarn hadoop   1180 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.9
-rw-r--r--  1 yarn hadoop   1126 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.8
-rw-r--r--  1 yarn hadoop   1184 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.7
-rw-r--r--  1 yarn hadoop   1184 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.6
-rw-r--r--  1 yarn hadoop   1088 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.5
-rw-r--r--  1 yarn hadoop   1158 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.4
-rw-r--r--  1 yarn hadoop   1042 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.3
-rw-r--r--  1 yarn hadoop   1147 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.2
-rw-r--r--  1 yarn hadoop   1531 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.out
-rw-r--r--  1 yarn hadoop   1090 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.1
-rw-r--r--  1 yarn hadoop 455502 Aug 27 12:06 rm-audit.log
-rw-r--r--  1 yarn hadoop    459 Aug 27 12:15 yarn-yarn-resourcemanager-master02.sys76.com.log
[root@master02 yarn]# pwd
/var/log/hadoop-yarn/yarn</strong>

its very strange because according to Advanced yarn-log4j from YARN config we have diff conf

we not understand why in /var/log/hadoop-yarn/yarn we get 10 backup files while yarn-log4j isn't define it

second each yarn-yarn-resourcemanager-master02.sys76.com.log.XX , should get 256M as define in Advanced yarn-log4j , but from looking under /var/log/hadoop-yarn/yarn , each backup is only 4K

so we not understand why each backup is 4K , and from where he get this value ?

Advanced yarn-log4j ( from YARN CONFIG )

#Relative to Yarn Log Dir Prefix
yarn.log.dir=.
#
# Job Summary Appender
#
# Use following logger to send summary to separate file defined by
# hadoop.mapreduce.jobsummary.log.file rolled daily:
# hadoop.mapreduce.jobsummary.logger=INFO,JSA
#
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log
log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender
# Set the ResourceManager summary log filename
yarn.server.resourcemanager.appsummary.log.file=hadoop-mapreduce.jobsummary.log
# Set the ResourceManager summary log level and appender
yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger}
#yarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY
# To enable AppSummaryLogging for the RM,
# set yarn.server.resourcemanager.appsummary.logger to
# LEVEL,RMSUMMARY in hadoop-env.sh
# Appender for ResourceManager Application Summary Log
# Requires the following properties to be set
#    - hadoop.log.dir (Hadoop Log directory)
#    - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename)
#    - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender)
log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender
log4j.appender.RMSUMMARY.File=${yarn.log.dir}/${yarn.server.resourcemanager.appsummary.log.file}
log4j.appender.RMSUMMARY.MaxFileSize=256MB
log4j.appender.RMSUMMARY.MaxBackupIndex=20
log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout
log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false
# Appender for viewing information for errors and warnings
yarn.ewma.cleanupInterval=300
yarn.ewma.messageAgeLimitSeconds=86400
yarn.ewma.maxUniqueMessages=250
log4j.appender.EWMA=org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
log4j.appender.EWMA.cleanupInterval=${yarn.ewma.cleanupInterval}
log4j.appender.EWMA.messageAgeLimitSeconds=${yarn.ewma.messageAgeLimitSeconds}
log4j.appender.EWMA.maxUniqueMessages=${yarn.ewma.maxUniqueMessages}
# Audit logging for ResourceManager
rm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
# Audit logging for NodeManager
nm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false
log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log
log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd

[#2 · Accepted solution]

@Michael Bronson

By default the NMAUDIT and RMAUDIT files will be using the "DailyRollingFileAppender" hence in a day the file size might vary and as soon as the date changes then Log will be rotated "rm-audit.log" and "nm-audit.log" so you might see file size varying.

If you want a fix size for log rotation then you will need RollingFileAppender (instead of DailyRollingFileAppender) as i see that you have already implemented the same on other thread. https://community.hortonworks.com/questions/214983/how-to-in-force-the-audit-logs-in-order-to-test-t...

    # Audit logging for ResourceManager
    rm.audit.logger=${hadoop.root.logger}
    log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
    log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
    #log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
    log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
    log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
    log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
    #log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
    log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
    log4j.appender.RMAUDIT.MaxBackupIndex=5
    log4j.appender.RMAUDIT.MaxFileSize=1MB

.

RollingFileAppender provides options like MaxFileSize to control the log rotation by file size.

[#3]

I have a similar question. How do i bring down the size of the audit logs inside HDFS? I have the yarn plugin enabled,ed for Ranger but no policies defined. Daily log size that is generating is around 12 gigs. I changed the debug mode to info still not useful. Where and how can I make the changes to advanced-yarn-log4j? I already have referred to https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h... didn't find it useful as there is no configuration for the advanced-yarn-log4j properties. Again, this is in HDFS. The logs are audit logs for YARN and we have no policies defined for YARN.

,

I have a question.

I want to control the size of teh logs being generated under HDFS audits. That is the Yarn audit logs that are saved in the HDFS. I have no pilicies setup in YARN and even then the average file size for the daily logs is near about 12 gigs. I want to bring down the size of this log. What changes do I make and where?

Again, this is for the audit logs getting saved in the HDFS. Ranger audit logs under /ranger/audit/yarn/