Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Manage logs HDP

Solved Go to solution

Manage logs HDP

Hello Everyone,

How can i manage all logs generate by HDP cluster? Because since i've been installing my cluster, i have more than 25GB of logs. It take more space in my file system and will be a problem for me. What's the best way to limit logs size?

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Manage logs HDP

Super Mentor

@Baruch AMOUSSOU DJANGBAN

Following article contains most of the details on how to control the logging and rolling of logs based on size:

How to control size of log files for various HDP components? https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h...

.

You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files:

maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB.

maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1.

.

View solution in original post

2 REPLIES 2
Highlighted

Re: Manage logs HDP

Super Mentor

@Baruch AMOUSSOU DJANGBAN

Following article contains most of the details on how to control the logging and rolling of logs based on size:

How to control size of log files for various HDP components? https://community.hortonworks.com/articles/8882/how-to-control-size-of-log-files-for-various-hdp-c.h...

.

You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files:

maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB.

maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1.

.

View solution in original post

Highlighted

Re: Manage logs HDP

Guru

Hi @Baruch AMOUSSOU DJANGBAN ,

to avoid running out of disk space, you should use some OS tools to let them manage handling of log files, e.g. logrotate. You can configure it to run on a daily basis, compressing log files and remove old ones. But this behaviour highly depends on your requirements, e.g. how long do you want to keep the logfiles on local nodes, ...

You can also check if you have DEBUG logging enabled in one of your services where you do not need it, which will increase the size of log output heavily.

If you want to collect the logs from all the nodes to one central location for muuuuch easier diving into issues, you could create a data flow e.g. ingesting all the log entries into Solr

HTH

Don't have an account?
Coming from Hortonworks? Activate your account here