Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to in force the audit logs in order to test the log rotation log4j configuration

avatar

hi all,

we configured the Audit logging for ResourceManager , ( in YARN --> Advanced yarn-log4j )

as the following:

# Audit logging for ResourceManager
rm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
#log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RMAUDIT.File=${yarn.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.RMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.RMAUDIT.MaxBackupIndex=5
log4j.appender.RMAUDIT.MaxFileSize=1MB

and

# Audit logging for NodeManager
nm.audit.logger=${hadoop.root.logger}
log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false
#log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.NMAUDIT.File=${yarn.log.dir}/nm-audit.log
log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
#log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd
log4j.appender.NMAUDIT=org.apache.log4j.RollingFileAppender
log4j.appender.NMAUDIT.MaxBackupIndex=5
log4j.appender.NMAUDIT.MaxFileSize=1MB




and restart the YARN service

now we want to see if rm-audit.log and nm-audit.log are rotated , but because we set 1MB in MaxFileSize

it will take time until logs will be rotated

-rw-r--r--  1 yarn hadoop    4180 Aug 28 09:32 rm-audit.log

is it possible to in force the logs to increase quickly to 1M and rotated as rm-audit.log.1 , and rm-audit.log.2 , etc

so we can sure that our log4j configuration is correctly ?

Michael-Bronson
1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar

@Jay what is the limit to set the variable - log4j.appender.NMAUDIT.MaxFileSize ?

for example can we set 1MB as log4j.appender.NMAUDIT.MaxFileSize=1MB

Michael-Bronson

avatar
Master Mentor

@Michael Bronson

Log4j Specification says you can set the value to KB/MB .. so 1MB is fine for testing. Later you can have little larger value based on your requirement.

The default value will be 10MB as per the specs: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxFileSize

Following ambari API call can trigger Yarn Service check multiple times to check the rm-audit.log growth.

# curl -sH "X-Requested-By: ambari" -u admin:admin -X POST -d '{"RequestInfo":{"context":"YARN Service Check","command":"YARN_SERVICE_CHECK"},"Requests/resource_filters":[{"service_name":"YARN"}]}'  http://hdfcluster1.example.com:8080/api/v1/clusters/TestCluster/requests

.

avatar

@Jay , second , do you think my configuration in log4j are corectly ?

Michael-Bronson

avatar
Master Mentor

@Michael Bronson

Yes, your configuration looks perfect.

I just changed the value of 1MB to 50KB and tested it on Resource Manager and it worked well

log4j.appender.RMAUDIT.MaxFileSize=50KB

.

Rotation worked. So it should work well with 1MB or any other higher size as well.


avatar

@Jay we run the service check but it fail on python time out , is any other idea to increase the logs ?

second

from where we get the - yarn-yarn-resourcemanager-.... files ?

they are not written in the log4j so I not understand how they create

-rw-r--r-- 1 yarn hadoop 1847 Aug 27 12:03 yarn-yarn-resourcemanager-master02.sys76.com.out.1
-rw-r--r-- 1 yarn hadoop 1052 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.10
-rw-r--r-- 1 yarn hadoop 1180 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.9
Michael-Bronson