question Re: What is a good resource for best practices around Hadoop log management? in Archives of Support Questions (Read Only)

What is a good resource for best practices around Hadoop log management?

rtempleton — Fri, 16 Sep 2022 09:47:21 GMT

Re: What is a good resource for best practices around Hadoop log management?

nsabharwal — Mon, 02 Nov 2015 22:27:00 GMT

It's not different than any typical RDBMS or Application log rotation policies. Based on the organization policies, logs can be rotated every 30 days or so to keep enough space on /var/log (Best practice is to have dedicated disk or partition of size ~ 200 GB)

Simple cron job can call a script to move logs to different location from /var/log to NFS or tape or external media.

This is helpful.

Re: What is a good resource for best practices around Hadoop log management?

rtempleton — Wed, 04 Nov 2015 07:12:40 GMT

Thanks @Neeraj - The 200GB guide line is something I can share with customers.

Re: What is a good resource for best practices around Hadoop log management?

nsabharwal — Wed, 04 Nov 2015 07:14:52 GMT

@Ryan Templeton

Yes. I am sure they will appreciate it

Re: What is a good resource for best practices around Hadoop log management?

mlochbihler — Thu, 05 Nov 2015 03:22:16 GMT

I second Ryan's comment on dedicated disk or partition for logs. that the biggest mistake made in the field is to not dedicate disk or partition to /var/log. it is not trivial to migrate logs if managed by Ambari after initial installation, although it is doable. If you fail into this situation, ask for instruction set from Hortonworks support. I had to do so and wished I had set this up initially on seperate disk