question How to monitor yarn applications actual memory usage in Archives of Support Questions (Read Only)

How to monitor yarn applications actual memory usage

ThomasLarsson — Fri, 02 Sep 2016 20:23:49 GMT

Hello.

I would like to monitor the actual memory usage of the yarn containers in our cluster. We are using defaults such as

mapreduce.map.memory.mb=X;

mapreduce.reduce.memory.mb=Y;

But if I have understood this correctly, these values are only used to determine the maximum limit for processes running inside the containers. Is it possible to get metrics out from yarn about the actual memory usage of the process that ran in a container?

It looks like something like this was implemented in https://issues.apache.org/jira/browse/YARN-2984 but I'm not sure how I can access that data.

Can you give me any tips regarding this?

Best Regards

/Thomas

Added:

I can see what I'm looking for in the nodemanager logs so I guess those logs could be harvested and analyzed. Any other tips

Example of nodemanager log:

2016-09-02 13:31:58,563 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 50811 for container-id container_e21_1472110676349_75100_01_006278: 668.7 MB of 2.5 GB physical memory used; 2.9 GB of 5.3 GB virtual memory used

Re: How to monitor yarn applications actual memory usage

cstanca — Sat, 03 Sep 2016 07:03:07 GMT

@Thomas Larsson

Resource Manager UI. You can get to it in two ways: http:/hostname:8088, where hostname is the host name of the server where Resource Manager service runs. Otherwise, from Ambari UI click on YARN (left bar) then click on Quick Links at top middle, then select Resource Manager. You will see the memory and CPU used for each container. One container is allocated to a task.

Good tutorial here: http://hadooptutorial.info/yarn-web-ui/

This is the visual. You could also build your own Grafana dashboard making calls to the REST API: https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Please don't forget to vote/accept best answer for your question.

Re: How to monitor yarn applications actual memory usage

sheetal — Fri, 18 Aug 2017 23:54:00 GMT

SmartSense provides this information (and much more than this) for every job.