number of active containers limited to 160 - cluster is never more than 30% utilized

[#1]

Hi All,

I have a cluster with 20 nodes with 2 distinct hardware configs so we are using config groups. Our problem is that even though we have available VCores and Mem we can never get more than 160 allocated containers. I hope the screen shot shows my issue - plenty of resources but only 160 allocated containers.

Any pointers would be appreciated.

[#2]

A little more info.

We are able to get beyond the 160 container limit by running multiple spark-shell with 100 executors each so perhaps this is limited to MR2.