Reply
Highlighted
Explorer
Posts: 11
Registered: ‎02-27-2019
Accepted Solution

Limit Kudu logs

We have already 10+ GB logs for Kudu Masters and 30+ GB logs for Kudu Tablet Servers. How to limit the number of log files to keep? The Kudu Configuration Reference mentiones only a limit on a log file size (--max_log_size), but no limit for the number of log files.

Master
Posts: 429
Registered: ‎07-01-2015

Re: Limit Kudu logs

Based on my observation there is no limit for max number of files, but Kudu runs a compression on old logs. So you can easily remove all the logs *.gz
Explorer
Posts: 11
Registered: ‎02-27-2019

Re: Limit Kudu logs

This is not so good, because a user has to reason about old logs. With dozens of tablet servers, it is necessary to use some automated configuration management (Ansible or similar tool).

Usually it is a function of the logging framework itself (like Log4J).

Master
Posts: 429
Registered: ‎07-01-2015

Re: Limit Kudu logs

Let me know if you find a configuration for this.
Cloudera Employee
Posts: 2
Registered: ‎10-15-2015

Re: Limit Kudu logs

If I understand correctly, you are talking about the logs in the configured --log_dir. By default Kudu will keep 10 log files per severity level. There is a flag to change that value, but it's currently marked as "experimental". It has been in Kudu for some time, so not changing it to stable is probably a bit of an oversight. I opened an Apache Kudu jira (KUDU-2754) to change it to a stable config. 

 

In the mean time, you can use the --max_log_files configuration by unlocking experimental configurations via --unlock_experimental_flags.