how to in force the audit logs in order to test the log rotation log4j configuration

[#1]

hi all,

we configured the Audit logging for ResourceManager , ( in YARN --> Advanced yarn-log4j )

as the following:

# Audit logging for ResourceManager
rm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
#log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.RMAUDIT.MaxBackupIndex=5
log4j.appender.RMAUDIT.MaxFileSize=1MB

and

# Audit logging for NodeManager
nm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false
#log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log
log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.NMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.NMAUDIT.MaxBackupIndex=5
log4j.appender.NMAUDIT.MaxFileSize=1MB

and restart the YARN service

now we want to see if rm-audit.log and nm-audit.log are rotated , but because we set 1MB in MaxFileSize

it will take time until logs will be rotated

-rw-r--r--  1 yarn hadoop    4180 Aug 28 09:32 rm-audit.log

is it possible to in force the logs to increase quickly to 1M and rotated as rm-audit.log.1 , and rm-audit.log.2 , etc

so we can sure that our log4j configuration is correctly ?

[#2 · Accepted solution]

@Michael Bronson

Please try running some "Yarn Service Checks" from ambari UI 10-20 times and keep checking the "Active Resource Manager" logs "rm-audit.log"

You can also try running the following ambari API call to trigger the Yarn Service check in a for look 20-50 times and monitor the audit log growth.

# curl -sH "X-Requested-By: ambari" -u admin:admin -X POST -d '{"RequestInfo":{"context":"YARN Service Check","command":"YARN_SERVICE_CHECK"},"Requests/resource_filters":[{"service_name":"YARN"}]}'  http://hdfcluster1.example.com:8080/api/v1/clusters/TestCluster/requests

.

[#3]

@Jay what is the limit to set the variable - log4j.appender.NMAUDIT.MaxFileSize ?

for example can we set 1MB as log4j.appender.NMAUDIT.MaxFileSize=1MB

[#4]

@Michael Bronson

Log4j Specification says you can set the value to KB/MB .. so 1MB is fine for testing. Later you can have little larger value based on your requirement.

The default value will be 10MB as per the specs: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxFileSize

Following ambari API call can trigger Yarn Service check multiple times to check the rm-audit.log growth.

# curl -sH "X-Requested-By: ambari" -u admin:admin -X POST -d '{"RequestInfo":{"context":"YARN Service Check","command":"YARN_SERVICE_CHECK"},"Requests/resource_filters":[{"service_name":"YARN"}]}'  http://hdfcluster1.example.com:8080/api/v1/clusters/TestCluster/requests

.

[#5]

@Jay , second , do you think my configuration in log4j are corectly ?

[#6]

@Michael Bronson

Yes, your configuration looks perfect.

I just changed the value of 1MB to 50KB and tested it on Resource Manager and it worked well

log4j.appender.RMAUDIT.MaxFileSize=50KB

.

Rotation worked. So it should work well with 1MB or any other higher size as well.

[#7]

@Jay we run the service check but it fail on python time out , is any other idea to increase the logs ?

second

from where we get the - yarn-yarn-resourcemanager-.... files ?

they are not written in the log4j so I not understand how they create

-rw-r--r-- 1 yarn hadoop 1847 Aug 27 12:03 yarn-yarn-resourcemanager-master02.sys76.com.out.1
-rw-r--r-- 1 yarn hadoop 1052 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.10
-rw-r--r-- 1 yarn hadoop 1180 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.9