Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Cloudera Community

Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.
Solved Go to solution

Advanced yarn-log4j not fit what we get in /var/log/hadoop-yarn/yarn

avatar
author-rank mike_bronson7
Guru

Created ‎08-27-2018 12:59 PM

hi all

under /var/log/hadoop-yarn/yarn we have the following logs


<strong> du -sh *
180K    hadoop-mapreduce.jobsummary.log
448K    rm-audit.log
132K    rm-audit.log.2018-08-01
8.0K    rm-audit.log.2018-08-02
4.0K    rm-audit.log.2018-08-03
4.0K    rm-audit.log.2018-08-06
8.0K    rm-audit.log.2018-08-09
4.0K    rm-audit.log.2018-08-12
8.0K    rm-audit.log.2018-08-13
4.0K    rm-audit.log.2018-08-17
36K     rm-audit.log.2018-08-18
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.1
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.10
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.2
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.3
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.4
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.5
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.6
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.7
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.8
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.log.9
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.1
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.2
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.3
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.4
4.0K    yarn-yarn-resourcemanager-master02.sys76.com.out.5</strong>


<strong>[root@master02 yarn]# \ls -ltr
total 904
-rw-r--r--. 1 yarn hadoop   2271 Aug  1 13:31 yarn-yarn-resourcemanager-master02.sys76.com.out.5
-rw-r--r--. 1 yarn hadoop 130412 Aug  1 23:58 rm-audit.log.2018-08-01
-rw-r--r--  1 yarn hadoop   2052 Aug  2 01:53 yarn-yarn-resourcemanager-master02.sys76.com.out.4
-rw-r--r--  1 yarn hadoop   1734 Aug  2 09:26 yarn-yarn-resourcemanager-master02.sys76.com.out.3
-rw-r--r--  1 yarn hadoop   8186 Aug  2 23:26 rm-audit.log.2018-08-02
-rw-r--r--  1 yarn hadoop    548 Aug  3 07:43 rm-audit.log.2018-08-03
-rw-r--r--  1 yarn hadoop    274 Aug  6 09:20 rm-audit.log.2018-08-06
-rw-r--r--  1 yarn hadoop   5480 Aug  9 15:36 rm-audit.log.2018-08-09
-rw-r--r--  1 yarn hadoop    274 Aug 12 16:29 rm-audit.log.2018-08-12
-rw-r--r--  1 yarn hadoop   6576 Aug 13 16:54 rm-audit.log.2018-08-13
-rw-r--r--  1 yarn hadoop   1370 Aug 17 23:56 rm-audit.log.2018-08-17
-rw-r--r--  1 yarn hadoop   1944 Aug 18 01:21 yarn-yarn-resourcemanager-master02.sys76.com.out.2
-rw-r--r--  1 yarn hadoop  34937 Aug 18 03:46 rm-audit.log.2018-08-18
-rw-r--r--. 1 yarn hadoop 176470 Aug 27 12:03 hadoop-mapreduce.jobsummary.log
-rw-r--r--  1 yarn hadoop   1847 Aug 27 12:03 yarn-yarn-resourcemanager-master02.sys76.com.out.1
-rw-r--r--  1 yarn hadoop   1052 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.10
-rw-r--r--  1 yarn hadoop   1180 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.9
-rw-r--r--  1 yarn hadoop   1126 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.8
-rw-r--r--  1 yarn hadoop   1184 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.7
-rw-r--r--  1 yarn hadoop   1184 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.6
-rw-r--r--  1 yarn hadoop   1088 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.5
-rw-r--r--  1 yarn hadoop   1158 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.4
-rw-r--r--  1 yarn hadoop   1042 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.3
-rw-r--r--  1 yarn hadoop   1147 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.2
-rw-r--r--  1 yarn hadoop   1531 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.out
-rw-r--r--  1 yarn hadoop   1090 Aug 27 12:06 yarn-yarn-resourcemanager-master02.sys76.com.log.1
-rw-r--r--  1 yarn hadoop 455502 Aug 27 12:06 rm-audit.log
-rw-r--r--  1 yarn hadoop    459 Aug 27 12:15 yarn-yarn-resourcemanager-master02.sys76.com.log
[root@master02 yarn]# pwd
/var/log/hadoop-yarn/yarn</strong>




its very strange because according to Advanced yarn-log4j from YARN config we have diff conf

we not understand why in /var/log/hadoop-yarn/yarn we get 10 backup files while yarn-log4j isn't define it

second each yarn-yarn-resourcemanager-master02.sys76.com.log.XX , should get 256M as define in Advanced yarn-log4j , but from looking under /var/log/hadoop-yarn/yarn , each backup is only 4K

so we not understand why each backup is 4K , and from where he get this value ?

Advanced yarn-log4j ( from YARN CONFIG )

#Relative to Yarn Log Dir Prefix
yarn.log.dir=.
#
# Job Summary Appender
#
# Use following logger to send summary to separate file defined by
# hadoop.mapreduce.jobsummary.log.file rolled daily:
# hadoop.mapreduce.jobsummary.logger=INFO,JSA
#
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log
log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender
# Set the ResourceManager summary log filename
yarn.server.resourcemanager.appsummary.log.file=hadoop-mapreduce.jobsummary.log
# Set the ResourceManager summary log level and appender
yarn.server.resourcemanager.appsummary.logger=${hadoop.root.logger}
#yarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY
# To enable AppSummaryLogging for the RM,
# set yarn.server.resourcemanager.appsummary.logger to
# LEVEL,RMSUMMARY in hadoop-env.sh
# Appender for ResourceManager Application Summary Log
# Requires the following properties to be set
#    - hadoop.log.dir (Hadoop Log directory)
#    - yarn.server.resourcemanager.appsummary.log.file (resource manager app summary log filename)
#    - yarn.server.resourcemanager.appsummary.logger (resource manager app summary log level and appender)
log4j.appender.RMSUMMARY=org.apache.log4j.RollingFileAppender
log4j.appender.RMSUMMARY.File=${yarn.log.dir}/${yarn.server.resourcemanager.appsummary.log.file}
log4j.appender.RMSUMMARY.MaxFileSize=256MB
log4j.appender.RMSUMMARY.MaxBackupIndex=20
log4j.appender.RMSUMMARY.layout=org.apache.log4j.PatternLayout
log4j.appender.RMSUMMARY.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=${yarn.server.resourcemanager.appsummary.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary=false
# Appender for viewing information for errors and warnings
yarn.ewma.cleanupInterval=300
yarn.ewma.messageAgeLimitSeconds=86400
yarn.ewma.maxUniqueMessages=250
log4j.appender.EWMA=org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender
log4j.appender.EWMA.cleanupInterval=${yarn.ewma.cleanupInterval}
log4j.appender.EWMA.messageAgeLimitSeconds=${yarn.ewma.messageAgeLimitSeconds}
log4j.appender.EWMA.maxUniqueMessages=${yarn.ewma.maxUniqueMessages}
# Audit logging for ResourceManager
rm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
# Audit logging for NodeManager
nm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false
log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log
log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd
Michael-Bronson
3,980 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank jsensharma author-rank
Master Mentor

Created ‎08-28-2018 11:49 AM

@Michael Bronson

By default the NMAUDIT and RMAUDIT files will be using the "DailyRollingFileAppender" hence in a day the file size might vary and as soon as the date changes then Log will be rotated "rm-audit.log" and "nm-audit.log" so you might see file size varying.

If you want a fix size for log rotation then you will need RollingFileAppender (instead of DailyRollingFileAppender) as i see that you have already implemented the same on other thread. https://community.hortonworks.com/questions/214983/how-to-in-force-the-audit-logs-in-order-to-test-t...

    # Audit logging for ResourceManager
    rm.audit.logger=${hadoop.root.logger}
    log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
    log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
    #log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
    log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
    log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
    log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
    #log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
    log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
    log4j.appender.RMAUDIT.MaxBackupIndex=5
    log4j.appender.RMAUDIT.MaxFileSize=1MB

.

RollingFileAppender provides options like MaxFileSize to control the log rotation by file size.

View solution in original post

3,608 Views
2 REPLIES 2

avatar
author-rank jsensharma author-rank
Master Mentor

Created ‎08-28-2018 11:49 AM

@Michael Bronson

By default the NMAUDIT and RMAUDIT files will be using the "DailyRollingFileAppender" hence in a day the file size might vary and as soon as the date changes then Log will be rotated "rm-audit.log" and "nm-audit.log" so you might see file size varying.

If you want a fix size for log rotation then you will need RollingFileAppender (instead of DailyRollingFileAppender) as i see that you have already implemented the same on other thread. https://community.hortonworks.com/questions/214983/how-to-in-force-the-audit-logs-in-order-to-test-t...

    # Audit logging for ResourceManager
    rm.audit.logger=${hadoop.root.logger}
    log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
    log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
    #log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
    log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
    log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
    log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
    #log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
    log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
    log4j.appender.RMAUDIT.MaxBackupIndex=5
    log4j.appender.RMAUDIT.MaxFileSize=1MB

.

RollingFileAppender provides options like MaxFileSize to control the log rotation by file size.

3,609 Views

avatar
author-rank ankur_gadgilwar
New Member

Created ‎10-16-2018 01:30 PM

I have a similar question. How do i bring down the size of the audit logs inside HDFS? I have the yarn plugin enabled,ed for Ranger but no policies defined. Daily log size that is generating is around 12 gigs. I changed the debug mode to info still not useful. Where and how can I make the changes to advanced-yarn-log4j? I already have referred to https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h... didn't find it useful as there is no configuration for the advanced-yarn-log4j properties. Again, this is in HDFS. The logs are audit logs for YARN and we have no policies defined for YARN.

,

I have a question.

I want to control the size of teh logs being generated under HDFS audits. That is the Yarn audit logs that are saved in the HDFS. I have no pilicies setup in YARN and even then the average file size for the daily logs is near about 12 gigs. I want to bring down the size of this log. What changes do I make and where?

Again, this is for the audit logs getting saved in the HDFS. Ranger audit logs under /ranger/audit/yarn/

3,608 Views
0 Kudos
Announcements
Community Announcements
Community Upgrade Announcement
What's New @ Cloudera
Announcing Cloudera Flow Management Operator for Kubernetes ...
What's New @ Cloudera
Announcing Cloudera Streams Messaging Operator for Kubernete...
Community Announcements
July 2026 Community Highlights
Community Announcements
June 2026 Community Highlights
View More Announcements