How to limit the size of ranger log and number of log files to retain?

[#1]

By default, Ranger uses the log4j DailyRollingFileAppender to manage rotation of log files. This is the appender used by many HDP components, and it does not have a concept of max number of files to keep around. The advantage of using this appender is that it is easier to locate a log entry since files are split up on date boundaries. The main disadvantage of this appender is that there is no way to limit the amount of log data that is kept.

There are two common options for dealing with this.

1. Keep the DailyRollingFileAppender, and trim logs using a cron script. The following script would only keep 30 days worth of logs:

#!/bin/bash

find /var/log/ranger -mtime +30 | xargs --no-run-if-empty rm

You can change the +30 to be +number_of_days.

This script could be installed, for example, in a file called /etc/cron.daily/trim_ranger_logs and it would run each morning and remove logs older than 30 days.

2. Change to the RollingFileAppender. To do this you need to modify the log4j.xml file for the appropriate component:

ranger-admin: /usr/hdp/<version>/ranger-admin/ews/webapp/WEB-INF/log4j.xml ranger-usersync:/etc/ranger/usersync/conf/log4j.xml

Change these files as follows:

1. Change the appender definitions which use org.apache.log4j.DailyRollingFileAppender to use org.apache.log4j.RollingFileAppender 2. Change those appenders additionally by removing the datePattern parameter, and replace it with two parameters:

maxBackupIndex - this would be set to the number of files you want to keep maxFileSize - this would be set the max size of each file before it's rotated.

For example, the xa_log_appenderfrom /usr/hdp/<version>/ranger-admin/ews/webapp/WEB-INF/log4j.xml suitably modified to rotate files after they grow to a size of 1MB and keep 30 of them looks like this:

<appender name="xa_log_appender">
<param name="file" value="${catalina.base}/logs/xa_portal.log" />
<param name="maxFileSize" value="1MB" />
<param name="maxBackupIndex" value="30" />
<param name="append" value="true" /> 
<layout class="org.apache.log4j.PatternLayout"> 
<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" /> </layout>
</appender>

Note that the default for maxFileSize is 10MB.

After you do this you need to restart the Ranger services.

The advantage to the RollingFileAppender is a predictable log footprint. The disadvantage is that your logs are no longer cleanly broken on date boundaries, which might lead you to spend more time searching your logs in case of a problem.

[#2]

Great content, thanks for sharing!