1. Kafka Kafka currently uses org.apache.log4j.DailyRollingFileAppender, which doesn’t allow us to specify the max backup index, max file size. And by default, rolls every hour creating 24 files a day. Appender: org.apache.log4j. RollingFileAppender Roll criteria: Every 100MB. Retention: 9 backup logs. Steps: Login to Ambari Goto the config tab of the Kafka broker. Expand Advanced kafka-log4j Change the appender from org.apache.log4j.DailyRollingFileAppender to org.apache.log4j.RollingFileAppender. Comment out the appender’s date pattern: #log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH #log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender Add the following lines for each appender (change the appender name): log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender log4j.appender.kafkaAppender.MaxFileSize=100MB log4j.appender.kafkaAppender.MaxBackupIndex=9 Repeat the steps for all the appenders in the section. Rolling restart Kafka brokers for the changes to take effect. Issues/Further Investigation: The kafka.out doesn’t roll and grows huge. There is a open ticket with HortonWorks on this issue. 2. Storm We need to set the log viewer settings to update the clean up frequency logviewer.cleanup.age.mins - Value in mins for the worker logs to age for cleaning. logviewer.cleanup.interval.secs - Value in secs for frequency for the logviewer cleanup process to trigger. Steps: Login to Ambari Goto the config tab of the Kafka broker. Expand Custom storm-site Add the following two properties: logviewer.cleanup.age.mins : XXX (Value in mins for the worker logs to age for cleaning) logviewer.cleanup.interval.secs : XXX (Value in secs for frequency for the logviewer cleanup process to trigger) 5. Restart Storm components. (esp.. Supervisors – rolling restart) 3. Ranger Ranger logs are divided into 2 parts: admin and usersync admin – Ranger admin logs – access logs, xa log and xa sql log usersync – User sync logs SSH to the server where Ranger Admin is installed and make the following changes: Admin logs : /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml Change that to RollingFileAppender, see example below: <appender name="xa_log_appender"> <param name="file" value="${catalina.base}/logs/xa_portal.log" /> <param name="datePattern" value="'.'yyyy-MM-dd" /> <param name="append" value="true" /> <layout> <param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" /> </layout> </appender> Change the above to: <appender name="xa_log_appender"> <param name="file" value="${catalina.base}/logs/xa_portal.log" /> <param name="maxFileSize" value="100MB" /> <param name="maxBackupIndex" value="9" /> <layout> <param name="ConversionPattern" value="%d [%t] %-5p %C{6} (%F:%L) - %m%n" /> </layout> </appender> Repeat the process for sql_appender as well. 2. UserSync logs (usr/hdp/current/ranger-usersync/conf/log4j.xml): Currently it uses DailyRollingFileAppender which doesn’t clean up the old files. <appender name="logFile"> <param name="file" value="${logdir}/usersync.log" /> <param name="DatePattern" value="'.'yyyy-MM-dd" /> <layout> <param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/> </layout> </appender> Change the above to: <appender name="logFile"> <param name="file" value="${logdir}/usersync.log" /> <param name="maxFileSize" value="100MB" /> <param name="maxBackupIndex" value="9" /> <layout> <param name="ConversionPattern" value="%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n"/> </layout> </appender> 3. Restart Ranger. 4. HDFS Audit Logs: This log is currently configured with DailyRollingFileAppender, have to change this to RollingFileAppender to enforce size based rolling and cleanup. Steps: Login to Ambari Goto the config tab of HDFS component. Filter for Advanced hdfs-log4j Update the following log appenders (changes are highlighted): #log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA=org.apache.log4j.RollingFileAppender ... # Rollver at midnight #log4j.appender.DRFA.DatePattern=.yyyy-MM-dd ... #log4j.appender.DRFA.MaxBackupIndex=30 log4j.appender.DRFA.MaxFileSize=256MB ... #log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAS=org.apache.log4j.RollingFileAppender log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file} log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n #log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd log4j.appender.DRFAS.MaxFileSize=100MB log4j.appender.DRFAS.MaxBackupIndex=9 hdfs.audit.logger=INFO,console log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger} log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false #log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n #log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.DRFAAUDIT.MaxFileSize=100MB log4j.appender.DRFAAUDIT.MaxBackupIndex=9 mapred.audit.logger=INFO,console log4j.logger.org.apache.hadoop.mapred.AuditLogger=${mapred.audit.logger} log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false #log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.MRAUDIT=org.apache.log4j.RollingFileAppender log4j.appender.MRAUDIT.File=${hadoop.log.dir}/mapred-audit.log log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n #log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd log4j.appender.MRAUDIT.MaxFileSize=100MB log4j.appender.MRAUDIT.MaxBackupIndex=9 5. ZooKeeper # Max log file size of 10MB log4j.appender.ROLLINGFILE.MaxFileSize=256MB # uncomment the next line to limit number of backup files #log4j.appender.ROLLINGFILE.MaxBackupIndex=10 6. Oozie Steps: Login to Ambari Goto the config tab of Oozie component. Filter for oozie.instance.id}] %m%n #Commented this to add RollingFileAppender #log4j.appender.oozieops=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieops.DatePattern='.'yyyy-MM-dd log4j.appender.oozieops=org.apache.log4j.RollingFileAppender log4j.appender.oozieops.MaxFileSize=256MB log4j.appender.oozieops.MaxBackupIndex=20 log4j.appender.oozieops.File=${oozie.log.dir}/oozie-ops.log log4j.appender.oozieops.Append=true log4j.appender.oozieops.layout=org.apache.log4j.PatternLayout log4j.appender.oozieops.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n #Commented this to add RollingFileAppender #log4j.appender.oozieinstrumentation=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieinstrumentation.DatePattern='.'yyyy-MM-dd log4j.appender.oozieinstrumentation=org.apache.log4j.RollingFileAppender log4j.appender.oozieinstrumentation.MaxFileSize=256MB log4j.appender.oozieinstrumentation.MaxBackupIndex=20 log4j.appender.oozieinstrumentation.File=${oozie.log.dir}/oozie-instrumentation.log log4j.appender.oozieinstrumentation.Append=true log4j.appender.oozieinstrumentation.layout=org.apache.log4j.PatternLayout log4j.appender.oozieinstrumentation.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n #Commented this to add RollingFileAppender #log4j.appender.oozieaudit=org.apache.log4j.DailyRollingFileAppender #log4j.appender.oozieaudit.DatePattern='.'yyyy-MM-dd log4j.appender.oozieaudit =org.apache.log4j.RollingFileAppender log4j.appender.oozieaudit.MaxFileSize=256MB log4j.appender.oozieaudit.MaxBackupIndex=20 log4j.appender.oozieaudit.File=${oozie.log.dir}/oozie-audit.log log4j.appender.oozieaudit.Append=true log4j.appender.oozieaudit.layout=org.apache.log4j.PatternLayout log4j.appender.oozieaudit.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n #Commented this to add RollingFileAppender #log4j.appender.openjpa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.openjpa.DatePattern='.'yyyy-MM-dd log4j.appender.openjpa=org.apache.log4j.RollingFileAppender log4j.appender.openjpa.MaxFileSize=256MB log4j.appender.openjpa.MaxBackupIndex=20 log4j.appender.openjpa.File=${oozie.log.dir}/oozie-jpa.log log4j.appender.openjpa.Append=true log4j.appender.openjpa.layout=org.apache.log4j.PatternLayout log4j.appender.openjpa.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n 7. Knox Login to Ambari Goto the config tab of Oozie component. Filter for gateway-log4j Update the following log appenders (changes are highlighted): #Commented this to add RollingFileAppender #log4j.rootLogger=ERROR, drfa log4j.rootLogger=ERROR, rfa log4j.logger.org.apache.hadoop.gateway=INFO #log4j.logger.org.apache.hadoop.gateway=DEBUG #log4j.logger.org.eclipse.jetty=DEBUG #log4j.logger.org.apache.shiro=DEBUG #log4j.logger.org.apache.http=DEBUG #log4j.logger.org.apache.http.client=DEBUG #log4j.logger.org.apache.http.headers=DEBUG #log4j.logger.org.apache.http.wire=DEBUG log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n #Commented this to add RollingFileAppender #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.drfa.File=${app.log.dir}/${app.log.file} #log4j.appender.drfa.DatePattern=.yyyy-MM-dd #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n log4j.appender.rfa=org.apache.log4j.RollingFileAppender log4j.appender.rfa.File=${app.log.dir}/${app.log.file} log4j.appender.rfa.MaxFileSize=256MB log4j.appender.rfa.MaxBackupIndex=10 log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n log4j.logger.audit=INFO, auditfile #Commented this to add RollingFileAppender #log4j.appender.auditfile=org.apache.log4j.DailyRollingFileAppender log4j.appender.auditfile=org.apache.log4j.RollingFileAppender log4j.appender.auditfile.MaxFileSize=256MB log4j.appender.auditfile.MaxBackupIndex=10 log4j.appender.auditfile.File=${app.log.dir}/${app.audit.file} log4j.appender.auditfile.Append = true #log4j.appender.auditfile.DatePattern = '.'yyyy-MM-dd log4j.appender.auditfile.layout = org.apache.hadoop.gateway.audit.log4j.layout.AuditLayout 5. Filter for ldap-log4j Update the following log appenders (changes are highlighted): #Commented this to add RollingFileAppender #log4j.rootLogger=ERROR, drfa log4j.rootLogger=ERROR, rfa log4j.logger.org.apache.directory.server.ldap.LdapServer=INFO log4j.logger.org.apache.directory=WARN log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n #Commented this to add RollingFileAppender #log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender #log4j.appender.drfa.File=${app.log.dir}/${app.log.file} #log4j.appender.drfa.DatePattern=.yyyy-MM-dd #log4j.appender.drfa.layout=org.apache.log4j.PatternLayout #log4j.appender.drfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n log4j.appender.rfa=org.apache.log4j.RollingFileAppender log4j.appender.rfa.File=${app.log.dir}/${app.log.file} log4j.appender.rfa.MaxFileSize=256MB log4j.appender.rfa.MaxBackupIndex=10 log4j.appender.rfa.layout=org.apache.log4j.PatternLayout log4j.appender.rfa.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n ... View more