What is a good resource for best practices around Hadoop log management?

[#1 · Accepted solution]

@Ryan Templeton

It's not different than any typical RDBMS or Application log rotation policies. Based on the organization policies, logs can be rotated every 30 days or so to keep enough space on /var/log (Best practice is to have dedicated disk or partition of size ~ 200 GB)

Simple cron job can call a script to move logs to different location from /var/log to NFS or tape or external media.

This is helpful.

[#2]

Thanks @Neeraj - The 200GB guide line is something I can share with customers.

[#3]

@Ryan Templeton

Yes. I am sure they will appreciate it

[#4]

I second Ryan's comment on dedicated disk or partition for logs. that the biggest mistake made in the field is to not dedicate disk or partition to /var/log. it is not trivial to migrate logs if managed by Ambari after initial installation, although it is doable. If you fail into this situation, ask for instruction set from Hortonworks support. I had to do so and wished I had set this up initially on seperate disk