How to control size of log files for various HDP components?

[#1]

1. Kafka

Kafka currently uses org.apache.log4j.DailyRollingFileAppender, which doesn’t allow us to specify the max backup index, max file size. And by default, rolls every hour creating 24 files a day.

Appender: org.apache.log4j. RollingFileAppender

Roll criteria: Every 100MB.

Retention: 9 backup logs.

Steps:

1. Login to Ambari
2. Goto the config tab of the Kafka broker.
3. Expand Advanced kafka-log4j
4. Change the appender from org.apache.log4j.DailyRollingFileAppender to org.apache.log4j.RollingFileAppender.
5. Comment out the appender’s date pattern:

   #log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
   	

   #log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
   	

6. Add the following lines for each appender (change the appender name):

   log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
   	

   log4j.appender.kafkaAppender.MaxFileSize=100MB
   	

   log4j.appender.kafkaAppender.MaxBackupIndex=9
   	

7. Repeat the steps for all the appenders in the section.
8. Rolling restart Kafka brokers for the changes to take effect.

Issues/Further Investigation:

The kafka.out doesn’t roll and grows huge. There is a open ticket with HortonWorks on this issue.

2. Storm

We need to set the log viewer settings to update the clean up frequency

logviewer.cleanup.age.mins - Value in mins for the worker logs to age for cleaning.

logviewer.cleanup.interval.secs - Value in secs for frequency for the logviewer cleanup process to trigger.

Steps:

1. Login to Ambari
2. Goto the config tab of the Kafka broker.
3. Expand Custom storm-site
4. Add the following two properties:

logviewer.cleanup.age.mins : XXX (Value in mins for the worker logs to age for cleaning)

logviewer.cleanup.interval.secs : XXX (Value in secs for frequency for the logviewer cleanup process to trigger)

5. Restart Storm components. (esp.. Supervisors – rolling restart)

3. Ranger

Ranger logs are divided into 2 parts: admin and usersync

admin – Ranger admin logs – access logs, xa log and xa sql log

usersync – User sync logs

SSH to the server where Ranger Admin is installed and make the following changes:

1. Admin logs : /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml

Change that to RollingFileAppender, see example below:

<appender name="xa_log_appender">

<param name="file" value="${catalina.base}/logs/xa_portal.log" />

<param name="datePattern" value="'.'yyyy-MM-dd" />

<param name="append" value="true" />

<layout>

<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />

</layout>

</appender>

Change the above to:

<appender name="xa_log_appender">

<param name="file" value="${catalina.base}/logs/xa_portal.log" />

<param name="maxFileSize" value="100MB" />

<param name="maxBackupIndex" value="9" />

<layout>

<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />

</layout>

</appender>

Repeat the process for sql_appender as well.

2. UserSync logs (usr/hdp/current/ranger-usersync/conf/log4j.xml):

Currently it uses DailyRollingFileAppender which doesn’t clean up the old files.

<appender name="logFile">

<param name="file" value="${logdir}/usersync.log" />

<param name="DatePattern" value="'.'yyyy-MM-dd" />

<layout>

<param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>

</layout>

</appender>

Change the above to:

<appender name="logFile">

<param name="file" value="${logdir}/usersync.log" />

<param name="maxFileSize" value="100MB" />

<param name="maxBackupIndex" value="9" />

<layout>

<param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>

</layout>

</appender>

3. Restart Ranger.

4. HDFS

Audit Logs:

This log is currently configured with DailyRollingFileAppender, have to change this to RollingFileAppender to enforce size based rolling and cleanup.

Steps:

1. Login to Ambari
2. Goto the config tab of HDFS component.
3. Filter for Advanced hdfs-log4j
4. Update the following log appenders (changes are highlighted):

#log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA=org.apache.log4j.RollingFileAppender

...

# Rollver at midnight #log4j.appender.DRFA.DatePattern=.yyyy-MM-dd

...

#log4j.appender.DRFA.MaxBackupIndex=30 log4j.appender.DRFA.MaxFileSize=256MB

...

#log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAS=org.apache.log4j.RollingFileAppender log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n #log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd log4j.appender.DRFAS.MaxFileSize=100MB log4j.appender.DRFAS.MaxBackupIndex=9

hdfs.audit.logger=INFO,console

log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}

log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false

#log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender

log4j.appender.DRFAAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log

log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

#log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

log4j.appender.DRFAAUDIT.MaxFileSize=100MB

log4j.appender.DRFAAUDIT.MaxBackupIndex=9

mapred.audit.logger=INFO,console

log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}

log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false

#log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender

log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log

log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

#log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd

log4j.appender.MRAUDIT.MaxFileSize=100MB

log4j.appender.MRAUDIT.MaxBackupIndex=9

5. ZooKeeper

# Max log file size of 10MB

log4j.appender.ROLLINGFILE.MaxFileSize=256MB # uncomment the next line to limit number of backup files #log4j.appender.ROLLINGFILE.MaxBackupIndex=10

6. Oozie

Steps:

1. Login to Ambari
2. Goto the config tab of Oozie component.
3. Filter for oozie.instance.id}] %m%n

   #Commented this to add RollingFileAppender #log4j.appender.oozieops=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieops.DatePattern='.'yyyy-MM-dd log4j.appender.oozieops=org.apache.log4j.RollingFileAppender log4j.appender.oozieops.MaxFileSize=256MB log4j.appender.oozieops.MaxBackupIndex=20 log4j.appender.oozieops.File=${oozie.log.dir}/oozie-ops.log log4j.appender.oozieops.Append=true log4j.appender.oozieops.layout=org.apache.log4j.PatternLayout log4j.appender.oozieops.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

   #Commented this to add RollingFileAppender #log4j.appender.oozieinstrumentation=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieinstrumentation.DatePattern='.'yyyy-MM-dd log4j.appender.oozieinstrumentation=org.apache.log4j.RollingFileAppender log4j.appender.oozieinstrumentation.MaxFileSize=256MB log4j.appender.oozieinstrumentation.MaxBackupIndex=20 log4j.appender.oozieinstrumentation.File=${oozie.log.dir}/oozie-instrumentation.log log4j.appender.oozieinstrumentation.Append=true log4j.appender.oozieinstrumentation.layout=org.apache.log4j.PatternLayout log4j.appender.oozieinstrumentation.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

   #Commented this to add RollingFileAppender #log4j.appender.oozieaudit=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieaudit.DatePattern='.'yyyy-MM-dd log4j.appender.oozieaudit =org.apache.log4j.RollingFileAppender log4j.appender.oozieaudit.MaxFileSize=256MB log4j.appender.oozieaudit.MaxBackupIndex=20 log4j.appender.oozieaudit.File=${oozie.log.dir}/oozie-audit.log log4j.appender.oozieaudit.Append=true log4j.appender.oozieaudit.layout=org.apache.log4j.PatternLayout log4j.appender.oozieaudit.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

   #Commented this to add RollingFileAppender #log4j.appender.openjpa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.openjpa.DatePattern='.'yyyy-MM-dd log4j.appender.openjpa=org.apache.log4j.RollingFileAppender log4j.appender.openjpa.MaxFileSize=256MB log4j.appender.openjpa.MaxBackupIndex=20 log4j.appender.openjpa.File=${oozie.log.dir}/oozie-jpa.log log4j.appender.openjpa.Append=true log4j.appender.openjpa.layout=org.apache.log4j.PatternLayout log4j.appender.openjpa.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

7. Knox

1. Login to Ambari
2. Goto the config tab of Oozie component.
3. Filter for gateway-log4j
4. Update the following log appenders (changes are highlighted):

#Commented this to add RollingFileAppender #log4j.rootLogger=ERROR, drfa log4j.rootLogger=ERROR, rfa

log4j.logger.org.apache.hadoop.gateway=INFO #log4j.logger.org.apache.hadoop.gateway=DEBUG

#log4j.logger.org.eclipse.jetty=DEBUG #log4j.logger.org.apache.shiro=DEBUG #log4j.logger.org.apache.http=DEBUG #log4j.logger.org.apache.http.client=DEBUG #log4j.logger.org.apache.http.headers=DEBUG #log4j.logger.org.apache.http.wire=DEBUG

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#Commented this to add RollingFileAppender #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.drfa.File=${app.log.dir}/${app.log.file} #log4j.appender.drfa.DatePattern=.yyyy-MM-dd #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

log4j.appender.rfa=org.apache.log4j.RollingFileAppender log4j.appender.rfa.File=${app.log.dir}/${app.log.file} log4j.appender.rfa.MaxFileSize=256MB log4j.appender.rfa.MaxBackupIndex=10 log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

log4j.logger.audit=INFO, auditfile #Commented this to add RollingFileAppender #log4j.appender.auditfile=org.apache.log4j.DailyRollingFileAppender log4j.appender.auditfile=org.apache.log4j.RollingFileAppender log4j.appender.auditfile.MaxFileSize=256MB log4j.appender.auditfile.MaxBackupIndex=10 log4j.appender.auditfile.File=${app.log.dir}/${app.audit.file} log4j.appender.auditfile.Append = true #log4j.appender.auditfile.DatePattern = '.'yyyy-MM-dd log4j.appender.auditfile.layout = org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout

5. Filter for ldap-log4j

1. Update the following log appenders (changes are highlighted):

   #Commented this to add RollingFileAppender #log4j.rootLogger=ERROR, drfa log4j.rootLogger=ERROR, rfa

   log4j.logger.org.apache.directory.server.ldap.LdapServer=INFO log4j.logger.org.apache.directory=WARN

   log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

   #Commented this to add RollingFileAppender #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.drfa.File=${app.log.dir}/${app.log.file} #log4j.appender.drfa.DatePattern=.yyyy-MM-dd #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

   log4j.appender.rfa=org.apache.log4j.RollingFileAppender log4j.appender.rfa.File=${app.log.dir}/${app.log.file} log4j.appender.rfa.MaxFileSize=256MB log4j.appender.rfa.MaxBackupIndex=10 log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

[#2]

Awesome!!!

[#3]

I know this is and old thread but can you also include instructions on how to change the log aggregation for Spark2?