Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive log rotation issue on CDH 6

avatar
Contributor

Hi community, 

 

I have found an issue on CDH 6 with hive log rotation. In this CDH version, hive log4j changes from rotation by size (configured by cloudera manager) to rotation by size and day (RFA to DRFA).

 

ISSUE

This change results in:

  1. Cloudera can not remove old logs (and the FS can be filled), property "Maximum Log File Backups" does not work properly.
  2. The rotated log name changed from index to dateIf we produced more logs than "Max Log Size", we only have one file per day, so we are overwriting this file with the new produced logs, losing the old logs on same day, and we will have X logs per day, but probably losing logs every day (huge problem when resolving issues).

 

I'm completely sure that is a new issue, because hive log4j configuration changes on this version.

 

SOLUTION

The solution for me was changing the "appender.DRFA.rollingPolicy.FileNamePattern" property through Cloudera Safety Valve in Hive Server 2,  "Logging Advanced Configuration Snippet". Adding this new line:

 

appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}.%i

After restarting Hiver Server 2, the log rotation behaviour changes, with this change, I only have extrictly "Maximum Log File Backups" files with "Max Log Size" size, keeping away that the FS can be filled.

 

DEBUGGING

This actions are only for debugging proposal

I recommend you to check if you are using this CDH version (in my case 6.0.1) and if you have this issue too, you can validate this behaviour generating more logs changing "Debug Level" to TRACE and reducing "Max Log Size Maximum" to 1MB and "Log File Backups" to 2 and run any query on hive.

 

If your new logs are not rotated properly (hive only keep 2 new logs), more probably you have the same issue.

 

On latest versions, Cloudera adds new properties on hive log4j, probably to avoid this issue, but it keeps rotation by day, so we can experiment the issue (point 2) too. I think that rotation by day pattern is not the best option, by index is more effective, even by day and hour (hour, minute and second).

 

Marc Casajus

 

@salimhussain @na 

1 ACCEPTED SOLUTION

avatar
Moderator

Update: this issue is fixed in CM7.2.1


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

Hello @marccasajus 

 

Yes, this has been documented internally as a BUG (OPSAPS-53043) and is currently not fixed.

 

 

Also, it looks you have already applied the changes which would address this.

avatar
Moderator

Update: this issue is fixed in CM7.2.1


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

avatar
Explorer

seems not working in CDH 6.2, this solution will cause hive2 restart failed. with  <No type attribute provided for component rollingPolicy>

hive log4j config in cdh6.2

---------------------------

property.max.log.file.size=200MB
property.max.log.file.backup.index=10

appender.DRFA.type=RollingRandomAccessFile
appender.DRFA.name=DRFA
appender.DRFA.fileName=${log.dir}/${log.file}
appender.DRFA.filePattern=${log.dir}/${log.file}.%d{yyyy-MM-dd}
appender.DRFA.layout.typetternLayout
appender.DRFA.layout.pattern=%d{DEFAULT} %-5p %c: [%t]: %m%n
appender.DRFA.policies.type=Policies
appender.DRFA.policies.size.type=SizeBasedTriggeringPolicy
appender.DRFA.policies.size.size=${max.log.file.size}
appender.DRFA.strategy.type=DefaultRolloverStrategy
appender.DRFA.strategy.max=${max.log.file.backup.index}

avatar
Explorer

it should be like this,  appender.DRFA.filePattern=${log.dir}/${log.file}.%d{yyyy-MM-dd}.%i

 

and it will be working in CDH 6.2

avatar
Explorer

tencentemr_0-1669356095392.png