We are running an 8 node virtualized cluster with 5 datanodes. Each datanode is allocated 8 vcores, 54 GB of RAM, and use shared SAN storage. The output of yarn-utils (v=8, m=54, d=4) is: yarn.scheduler.minimum-allocation-mb=6656 yarn.scheduler.maximum-allocation-mb=53248 yarn.nodemanager.resource.memory-mb=53248 mapreduce.map.memory.mb=6656 mapreduce.map.java.opts=-Xmx5324m mapreduce.reduce.memory.mb=6656
mapreduce.reduce.java.opts=-Xmx5324m yarn.app.mapreduce.am.resource.mb=6656 yarn.app.mapreduce.am.command-opts=-Xmx5324m mapreduce.task.io.sort.mb=2662 Some questions I have is 1) what do you put for disks value when data node disks are running on shared SAN storage? and; 2) The maximum container size only shows 8 GB even though each node is assigned 54 GB. Does this have something to do with over commitment in the virtual environment? yarn-utils wants it set to 53 GB.
... View more
