Community Articles

Find and share helpful community-sourced technical articles.
avatar
New Contributor

1. Kafka

Kafka currently uses org.apache.log4j.DailyRollingFileAppender, which doesn’t allow us to specify the max backup index, max file size. And by default, rolls every hour creating 24 files a day.

Appender: org.apache.log4j. RollingFileAppender

Roll criteria: Every 100MB.

Retention: 9 backup logs.

Steps:

  1. Login to Ambari
  2. Goto the config tab of the Kafka broker.
  3. Expand Advanced kafka-log4j
  4. Change the appender from org.apache.log4j.DailyRollingFileAppender to org.apache.log4j.RollingFileAppender.
  5. Comment out the appender’s date pattern:
    #log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
    	
    #log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
    	
  6. Add the following lines for each appender (change the appender name):
    log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
    	
    log4j.appender.kafkaAppender.MaxFileSize=100MB
    	
    log4j.appender.kafkaAppender.MaxBackupIndex=9
    	
  7. Repeat the steps for all the appenders in the section.
  8. Rolling restart Kafka brokers for the changes to take effect.

Issues/Further Investigation:

The kafka.out doesn’t roll and grows huge. There is a open ticket with HortonWorks on this issue.

2. Storm

We need to set the log viewer settings to update the clean up frequency

logviewer.cleanup.age.mins - Value in mins for the worker logs to age for cleaning.

logviewer.cleanup.interval.secs - Value in secs for frequency for the logviewer cleanup process to trigger.

Steps:

  1. Login to Ambari
  2. Goto the config tab of the Kafka broker.
  3. Expand Custom storm-site
  4. Add the following two properties:
logviewer.cleanup.age.mins : XXX (Value in mins for the worker logs to age for cleaning)
logviewer.cleanup.interval.secs : XXX (Value in secs for frequency for the logviewer cleanup process to trigger)

5. Restart Storm components. (esp.. Supervisors – rolling restart)

3. Ranger

Ranger logs are divided into 2 parts: admin and usersync

admin – Ranger admin logs – access logs, xa log and xa sql log

usersync – User sync logs

SSH to the server where Ranger Admin is installed and make the following changes:

  1. Admin logs : /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml

Change that to RollingFileAppender, see example below:

<appender name="xa_log_appender">

<param name="file" value="${catalina.base}/logs/xa_portal.log" />

<param name="datePattern" value="'.'yyyy-MM-dd" />

<param name="append" value="true" />

<layout>

<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />

</layout>

</appender>

Change the above to:

<appender name="xa_log_appender">

<param name="file" value="${catalina.base}/logs/xa_portal.log" />

<param name="maxFileSize" value="100MB" />

<param name="maxBackupIndex" value="9" />

<layout>

<param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" />

</layout>

</appender>

Repeat the process for sql_appender as well.

2. UserSync logs (usr/hdp/current/ranger-usersync/conf/log4j.xml):

Currently it uses DailyRollingFileAppender which doesn’t clean up the old files.

<appender name="logFile">

<param name="file" value="${logdir}/usersync.log" />

<param name="DatePattern" value="'.'yyyy-MM-dd" />

<layout>

<param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>

</layout>

</appender>

Change the above to:

<appender name="logFile">

<param name="file" value="${logdir}/usersync.log" />

<param name="maxFileSize" value="100MB" />

<param name="maxBackupIndex" value="9" />

<layout>

<param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/>

</layout>

</appender>

3. Restart Ranger.

4. HDFS

Audit Logs:

This log is currently configured with DailyRollingFileAppender, have to change this to RollingFileAppender to enforce size based rolling and cleanup.

Steps:

  1. Login to Ambari
  2. Goto the config tab of HDFS component.
  3. Filter for Advanced hdfs-log4j
  4. Update the following log appenders (changes are highlighted):

#log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA=org.apache.log4j.RollingFileAppender

...

# Rollver at midnight #log4j.appender.DRFA.DatePattern=.yyyy-MM-dd

...

#log4j.appender.DRFA.MaxBackupIndex=30 log4j.appender.DRFA.MaxFileSize=256MB

...

#log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAS=org.apache.log4j.RollingFileAppender log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n #log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd log4j.appender.DRFAS.MaxFileSize=100MB log4j.appender.DRFAS.MaxBackupIndex=9

hdfs.audit.logger=INFO,console

log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}

log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false

#log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender

log4j.appender.DRFAAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log

log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

#log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

log4j.appender.DRFAAUDIT.MaxFileSize=100MB

log4j.appender.DRFAAUDIT.MaxBackupIndex=9

mapred.audit.logger=INFO,console

log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger}

log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false

#log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender

log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender

log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log

log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout

log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n

#log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd

log4j.appender.MRAUDIT.MaxFileSize=100MB

log4j.appender.MRAUDIT.MaxBackupIndex=9

5. ZooKeeper

# Max log file size of 10MB

log4j.appender.ROLLINGFILE.MaxFileSize=256MB # uncomment the next line to limit number of backup files #log4j.appender.ROLLINGFILE.MaxBackupIndex=10

6. Oozie

Steps:

  1. Login to Ambari
  2. Goto the config tab of Oozie component.
  3. Filter for oozie.instance.id}] %m%n

    #Commented this to add RollingFileAppender #log4j.appender.oozieops=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieops.DatePattern='.'yyyy-MM-dd log4j.appender.oozieops=org.apache.log4j.RollingFileAppender log4j.appender.oozieops.MaxFileSize=256MB log4j.appender.oozieops.MaxBackupIndex=20 log4j.appender.oozieops.File=${oozie.log.dir}/oozie-ops.log log4j.appender.oozieops.Append=true log4j.appender.oozieops.layout=org.apache.log4j.PatternLayout log4j.appender.oozieops.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

    #Commented this to add RollingFileAppender #log4j.appender.oozieinstrumentation=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieinstrumentation.DatePattern='.'yyyy-MM-dd log4j.appender.oozieinstrumentation=org.apache.log4j.RollingFileAppender log4j.appender.oozieinstrumentation.MaxFileSize=256MB log4j.appender.oozieinstrumentation.MaxBackupIndex=20 log4j.appender.oozieinstrumentation.File=${oozie.log.dir}/oozie-instrumentation.log log4j.appender.oozieinstrumentation.Append=true log4j.appender.oozieinstrumentation.layout=org.apache.log4j.PatternLayout log4j.appender.oozieinstrumentation.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

    #Commented this to add RollingFileAppender #log4j.appender.oozieaudit=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieaudit.DatePattern='.'yyyy-MM-dd log4j.appender.oozieaudit =org.apache.log4j.RollingFileAppender log4j.appender.oozieaudit.MaxFileSize=256MB log4j.appender.oozieaudit.MaxBackupIndex=20 log4j.appender.oozieaudit.File=${oozie.log.dir}/oozie-audit.log log4j.appender.oozieaudit.Append=true log4j.appender.oozieaudit.layout=org.apache.log4j.PatternLayout log4j.appender.oozieaudit.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

    #Commented this to add RollingFileAppender #log4j.appender.openjpa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.openjpa.DatePattern='.'yyyy-MM-dd log4j.appender.openjpa=org.apache.log4j.RollingFileAppender log4j.appender.openjpa.MaxFileSize=256MB log4j.appender.openjpa.MaxBackupIndex=20 log4j.appender.openjpa.File=${oozie.log.dir}/oozie-jpa.log log4j.appender.openjpa.Append=true log4j.appender.openjpa.layout=org.apache.log4j.PatternLayout log4j.appender.openjpa.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n

7. Knox

  1. Login to Ambari
  2. Goto the config tab of Oozie component.
  3. Filter for gateway-log4j
  4. Update the following log appenders (changes are highlighted):

#Commented this to add RollingFileAppender #log4j.rootLogger=ERROR, drfa log4j.rootLogger=ERROR, rfa

log4j.logger.org.apache.hadoop.gateway=INFO #log4j.logger.org.apache.hadoop.gateway=DEBUG

#log4j.logger.org.eclipse.jetty=DEBUG #log4j.logger.org.apache.shiro=DEBUG #log4j.logger.org.apache.http=DEBUG #log4j.logger.org.apache.http.client=DEBUG #log4j.logger.org.apache.http.headers=DEBUG #log4j.logger.org.apache.http.wire=DEBUG

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#Commented this to add RollingFileAppender #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.drfa.File=${app.log.dir}/${app.log.file} #log4j.appender.drfa.DatePattern=.yyyy-MM-dd #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

log4j.appender.rfa=org.apache.log4j.RollingFileAppender log4j.appender.rfa.File=${app.log.dir}/${app.log.file} log4j.appender.rfa.MaxFileSize=256MB log4j.appender.rfa.MaxBackupIndex=10 log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

log4j.logger.audit=INFO, auditfile #Commented this to add RollingFileAppender #log4j.appender.auditfile=org.apache.log4j.DailyRollingFileAppender log4j.appender.auditfile=org.apache.log4j.RollingFileAppender log4j.appender.auditfile.MaxFileSize=256MB log4j.appender.auditfile.MaxBackupIndex=10 log4j.appender.auditfile.File=${app.log.dir}/${app.audit.file} log4j.appender.auditfile.Append = true #log4j.appender.auditfile.DatePattern = '.'yyyy-MM-dd log4j.appender.auditfile.layout = org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout

5. Filter for ldap-log4j

  1. Update the following log appenders (changes are highlighted):

    #Commented this to add RollingFileAppender #log4j.rootLogger=ERROR, drfa log4j.rootLogger=ERROR, rfa

    log4j.logger.org.apache.directory.server.ldap.LdapServer=INFO log4j.logger.org.apache.directory=WARN

    log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

    #Commented this to add RollingFileAppender #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.drfa.File=${app.log.dir}/${app.log.file} #log4j.appender.drfa.DatePattern=.yyyy-MM-dd #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

    log4j.appender.rfa=org.apache.log4j.RollingFileAppender log4j.appender.rfa.File=${app.log.dir}/${app.log.file} log4j.appender.rfa.MaxFileSize=256MB log4j.appender.rfa.MaxBackupIndex=10 log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

35,763 Views
Comments

Awesome!!!

I know this is and old thread but can you also include instructions on how to change the log aggregation for Spark2?