Hello guys, i don't get YARN, i set all the required stuff like yarn.nodemanager.resource.memory-mb, yarn.scheduler.resource... ecc. But i see that the Docker container created by YARN (for AM or Executor) is always the total size of the Server (in my case 23GB is the total memory size of my server).
This of course leads to GREEDY application (like all applications, that if they can have more resources they will use it) and so i arrive to the point that the YARN limit that is passed and YARN kills my application saying "is running 112234496B beyond the 'PHYSICAL' memory limit. Current usage: 12.2 GB of 12.1 GB physical memory used; 35.0 GB of 25.5 GB virtual memory used. Killing container." (Where 12.1 is the limit set to the Queue where my app runs)
How can i tell YARN "YOU SHOULD BE THE ONE PREVENTING THIS" ? Since i can't find parameters for Docker called by YARN (a simply --memory parameter during the run command would be enough).
It's so annoying!!! I'm becoming mad.
P.S. i use ResourceManager with DominantResourceCalculator and submit to yarn via spark-submit.
Thanks!
