After about 2 weeks of various tries we finally settled on a full wipe of every host for a clean install from scratch. Still nothing working. Then we tried a "one worker" setup to set a countable resource manually to try the allocation mechanism and then.... NOTHING hortonWORKS ! But my Googling was better suited then. It seems to be a Hadoop related issue about custom resources and CapacityScheduler, enjoy: https://issues.apache.org/jira/browse/YARN-9161 https://issues.apache.org/jira/browse/YARN-9205 Temporary solution to benefit from isolation: For now (3.1.1/3.2.0) the capacity.CapacityScheduler is broken by a hardcoded enum containing only vCores and RAM parameters. You just have to switch your scheduler class to org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler You also want to replace "capacity" by "Fair" in the line yarn.scheduler.fair.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator Your GPUs will not be visible on yarn ui2 but will still be on the NodeManagers, and most importantly, will be allocated properly. It was a mess to find out indeed.
... View more
