Support Questions
Find answers, ask questions, and share your expertise

What is the recommended way to find the average time per day for which YARN Memory utilization is greater than X%

What is the recommended way to find the average time per day for which YARN Memory utilization is greater than X%

Contributor

Hi All,

This is a more capacity planning question. Over the last few months i have observed that YARN RAM utilization of our cluster has gone up considerably , due to new process being on boarded onto the cluster. At this point i am trying to find out the length of time (cumulative) through the day for which RAM usage is peaking.

Any information on how to go about this will be helpfull.

Currently:

  1. Hadoop2.7.1.2.4.0.0-169
  2. Ambari - 2.2.1
  3. HDP 2.4
  4. Master - 2 (63 GB RAM) ,DN -6 (63 GB RAM)
  5. RAM 320 GB Total allocation.
1 REPLY 1

Re: What is the recommended way to find the average time per day for which YARN Memory utilization is greater than X%

Explorer

@Suhel , You can try to get report from Ambari Metrics. or another way is to schedule a job to collect memory data on every machine after every 60 seconds. You can use "memfree" or "sar" command to collect memory usage with timestamp and keep appending output to a file and generate report top on that in excel or any other reporting tool to find out memory usage per node.

Regards,

Fahim