Manage logs HDP

[#1]

Hello Everyone,

How can i manage all logs generate by HDP cluster? Because since i've been installing my cluster, i have more than 25GB of logs. It take more space in my file system and will be a problem for me. What's the best way to limit logs size?

Thanks

[#2 · Accepted solution]

@Baruch AMOUSSOU DJANGBAN

Following article contains most of the details on how to control the logging and rolling of logs based on size:

How to control size of log files for various HDP components? https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h...

.

You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files:

maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB.

maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1.

.

[#3]

Hi @Baruch AMOUSSOU DJANGBAN ,

to avoid running out of disk space, you should use some OS tools to let them manage handling of log files, e.g. logrotate. You can configure it to run on a daily basis, compressing log files and remove old ones. But this behaviour highly depends on your requirements, e.g. how long do you want to keep the logfiles on local nodes, ...

You can also check if you have DEBUG logging enabled in one of your services where you do not need it, which will increase the size of log output heavily.

If you want to collect the logs from all the nodes to one central location for muuuuch easier diving into issues, you could create a data flow e.g. ingesting all the log entries into Solr

HTH