I would like to monitor the actual memory usage of the yarn containers in our cluster. We are using defaults such as
But if I have understood this correctly, these values are only used to determine the maximum limit for processes running inside the containers. Is it possible to get metrics out from yarn about the actual memory usage of the process that ran in a container?
It looks like something like this was implemented in https://issues.apache.org/jira/browse/YARN-2984 but I'm not sure how I can access that data.
Can you give me any tips regarding this?
I can see what I'm looking for in the nodemanager logs so I guess those logs could be harvested and analyzed. Any other tips
Example of nodemanager log:
2016-09-02 13:31:58,563 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 50811 for container-id container_e21_1472110676349_75100_01_006278: 668.7 MB of 2.5 GB physical memory used; 2.9 GB of 5.3 GB virtual memory used