I have a cluster with 20 nodes with 2 distinct hardware configs so we are using config groups. Our problem is that even though we have available VCores and Mem we can never get more than 160 allocated containers. I hope the screen shot shows my issue - plenty of resources but only 160 allocated containers.
Any pointers would be appreciated.
A little more info.
We are able to get beyond the 160 container limit by running multiple spark-shell with 100 executors each so perhaps this is limited to MR2.