Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to monitor yarn applications actual memory usage

avatar
Rising Star

Hello.

I would like to monitor the actual memory usage of the yarn containers in our cluster. We are using defaults such as

mapreduce.map.memory.mb=X;

mapreduce.reduce.memory.mb=Y;

But if I have understood this correctly, these values are only used to determine the maximum limit for processes running inside the containers. Is it possible to get metrics out from yarn about the actual memory usage of the process that ran in a container?

It looks like something like this was implemented in https://issues.apache.org/jira/browse/YARN-2984 but I'm not sure how I can access that data.

Can you give me any tips regarding this?

Best Regards

/Thomas

Added:

I can see what I'm looking for in the nodemanager logs so I guess those logs could be harvested and analyzed. Any other tips

Example of nodemanager log:

2016-09-02 13:31:58,563 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 50811 for container-id container_e21_1472110676349_75100_01_006278: 668.7 MB of 2.5 GB physical memory used; 2.9 GB of 5.3 GB virtual memory used
1 ACCEPTED SOLUTION

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Rising Star

SmartSense provides this information (and much more than this) for every job.