Hive log rotation issue on CDH 6
Hi community,
I have found an issue on CDH 6 with hive log rotation. In this CDH version, hive log4j changes from rotation by size (configured by cloudera manager) to rotation by size and day (RFA to DRFA).
ISSUE
This change results in:
- Cloudera can not remove old logs (and the FS can be filled), property "Maximum Log File Backups" does not work properly.
- The rotated log name changed from index to date, If we produced more logs than "Max Log Size", we only have one file per day, so we are overwriting this file with the new produced logs, losing the old logs on same day, and we will have X logs per day, but probably losing logs every day (huge problem when resolving issues).
I'm completely sure that is a new issue, because hive log4j configuration changes on this version.
SOLUTION
The solution for me was changing the "appender.DRFA.rollingPolicy.FileNamePattern" property through Cloudera Safety Valve in Hive Server 2, "Logging Advanced Configuration Snippet". Adding this new line:
appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}.%i
After restarting Hiver Server 2, the log rotation behaviour changes, with this change, I only have extrictly "Maximum Log File Backups" files with "Max Log Size" size, keeping away that the FS can be filled.
DEBUGGING
This actions are only for debugging proposal
I recommend you to check if you are using this CDH version (in my case 6.0.1) and if you have this issue too, you can validate this behaviour generating more logs changing "Debug Level" to TRACE and reducing "Max Log Size Maximum" to 1MB and "Log File Backups" to 2 and run any query on hive.
If your new logs are not rotated properly (hive only keep 2 new logs), more probably you have the same issue.
On latest versions, Cloudera adds new properties on hive log4j, probably to avoid this issue, but it keeps rotation by day, so we can experiment the issue (point 2) too. I think that rotation by day pattern is not the best option, by index is more effective, even by day and hour (hour, minute and second).
Marc Casajus
