The container memory usage limits are driven not by the available host memory but by the resource limits applied by the container configuration. For example if you've configured a map task to use 1 GiB of pmem, but its actual code at runtime uses more than 1 GiB, it will get killed. The common resolution to this would be to grant it more than 1 GiB, so it may do its higher-memory work without exceeding what it is given. Another resolution in certain cases would be to investigate if the excess memory use is justified, which can be discussed with the developer of the application. The randomness may be dependent on the amount of data the container code processes and what it ends up doing with it. Have you tried increasing the memory properties of containers via fields such as "Map Task Memory", "Reduce Task Memory" if its MR jobs you are having issues with, or pass higher values to --executor-memory arguments with spark-submit if its Spark jobs instead. This is all assuming you are seeing an error of the below form, since the relevant log isn't shared in your post: … Container killed by YARN for exceeding memory limits. 1.1 GB of 1 GB physical memory used …
... View more
