04-07-2016 01:50 PM
We do not expose the vmem setting in Cloudera Manager since it is really troublesome to get that check correct. Depending on how the memory gets allocated the virtual memory overhead could be anywhere between 5% of the JVM size to multiple times the full JVM size.
Your container is getting killed due to the physical memory (not virtual memory) over use. the best thing is to make sure that you allow for an overhead on top of the JVM size. We normally recommend 20% of the JVM heap size as the overhead. Again this is workload dependent and could differ for you.
We are working on the change that you only need to set one of the two and fully support that in Cloudera Manager. Some of the changes have been made to the underlying MR code already via MAPREDUCE-5785...
04-06-2018 01:05 AM
This does not seem to have worked with a latter version of CDH (5.13.1). There we had to set this through -
YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml
So, what is the correct way to set this? Is this really changed with newer releases?