Let's clarify some confusion - we're not talking about Kafka data logs, but rather logging of the Kafka broker process itself. So, your logs are getting big? Several solutions, based on your appetite for coding and linux admin automation.
Reduce Logging Output to WARN
Kafka broker is quite chatty about client connections, and that fills up logs quickly. Update the logger level for 'server.log' to write only WARN and above. E.g. in Ambari, go to Kafka -> Config -> Advanced kafka-log4j section. Scroll down and find the log4j.logger.kafka entry, modify the level to be WARN:
Change hourly logs to daily and rotate
By default, those logs in question will create hourly files. If you pair it with some external rotation/deletion policy, might want to switch to daily logs instead of hourly. In the same section above, find and update the log4j.appender.kafkaAppender.DatePattern . See the reference docs for syntax: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
Use LogRotate
Research http://linuxcommand.org/man_pages/logrotate8.html . Powerful, but given that you already have log4j in Kafka, might be redundant. An option if you want to go more on the admin side than app dev/ops.
Use an enhanced Log4j rolling appender with MaxBackupIndex
The version of log4j shipped by default doesn't support MaxBackupIndex attribute on the DailyRollingFileAppender. You can, however, find it in many libraries on the internet or quickly compile it yourself from e.g. this: http://wiki.apache.org/logging-log4j/DailyRollingFileAppender . Once you drop the extra jar in Kafka's lib directory, you could add the log4j.logger.kafka.MaxBackupIndex attribute in the config to specify how many of those log files to keep around.
WARNING: using the MaxBackupIndex also means logs will be lost if not picked up in time.
Let's clarify some confusion - we're not talking about Kafka data logs, but rather logging of the Kafka broker process itself. So, your logs are getting big? Several solutions, based on your appetite for coding and linux admin automation.
Reduce Logging Output to WARN
Kafka broker is quite chatty about client connections, and that fills up logs quickly. Update the logger level for 'server.log' to write only WARN and above. E.g. in Ambari, go to Kafka -> Config -> Advanced kafka-log4j section. Scroll down and find the log4j.logger.kafka entry, modify the level to be WARN:
Change hourly logs to daily and rotate
By default, those logs in question will create hourly files. If you pair it with some external rotation/deletion policy, might want to switch to daily logs instead of hourly. In the same section above, find and update the log4j.appender.kafkaAppender.DatePattern . See the reference docs for syntax: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
Use LogRotate
Research http://linuxcommand.org/man_pages/logrotate8.html . Powerful, but given that you already have log4j in Kafka, might be redundant. An option if you want to go more on the admin side than app dev/ops.
Use an enhanced Log4j rolling appender with MaxBackupIndex
The version of log4j shipped by default doesn't support MaxBackupIndex attribute on the DailyRollingFileAppender. You can, however, find it in many libraries on the internet or quickly compile it yourself from e.g. this: http://wiki.apache.org/logging-log4j/DailyRollingFileAppender . Once you drop the extra jar in Kafka's lib directory, you could add the log4j.logger.kafka.MaxBackupIndex attribute in the config to specify how many of those log files to keep around.
WARNING: using the MaxBackupIndex also means logs will be lost if not picked up in time.
