HDFS Audit Log File Size Issues

[#1]

Hi All,

HDFS Audit logs size have become huge since recent past days, on weekends /root was completely occupied, upon checking largest files in /root it was hdfs.audit.log file which were sizing between 300 MB to 9.9 GB. When I checked file sizes from Oldest to Newest, oldest were 300 MB- 500MB sized, the Newest which are from last 6 days were sizing between 1.9 GB to 9.9 GB! Are there any particular reason for huge space occupancy all of a sudden?

Ps: We have not installed Ranger, KNOX, SOLR and ATLAS.

[#2 · Accepted solution]

You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files:

maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB.

maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1.

More details in :

https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h...

[#3]

This posts also explains similar issue:

https://community.hortonworks.com/questions/78699/how-to-rotate-and-archive-hdfs-audit-log-file.html

[#4]

Thanks Namit, this worked for me on my Dev Environment. Will try on next Change on Prod too. Thanks.