Impala admission control memory issues with impala dedicates coordinator

[#1]

Hello Team,

Following is our Impala cluster config

•  Two dedicated executors with following hardware config 128GB, 16 core.
• One dedicated coordinator with following hardware config 8GB , 2 core.

As most of the query processing happens on the executor we have kept the executor's size more and co-ordinator's size small.

Now when we enable Impala admission controls we run into the situation where queries having memory requirements more than that of coordinator's memory gets rejected because of insufficient memory error, even though the cluster on the whole has enough memory to process the request. This is the error message which I get 

Rejected query from pool root.default: request memory needed 8.29 GB per node is greater than memory available for admission 6.00 GB of <coordinator's DNS>. Use the MEM_LIMIT query option to indicate how much memory is required per node.

Following is the admission control config

As of now we only one default pool

Max Memory :- 200GB

Default Memory Limit :- No default limit

Max queued queries :- 200

Queue Timeout :- 5 Minutes 

What can be the reasons for this error? Does executor's and coordinator's hardware size need to be same? Can I exclude coordinator from the admission control?

Regards

Parth 

[#2]

@EricL  can you help me with this.

[#3]

We are using Impala version 3.2.0.

[#4]

In Impala in CDH6, queries reserve the same amount of memory on the coordinator as on the executors. I.e. coordinators need to be given enough memory so that queries can reserve the same amount on the coordinator as executors. So the config with 1 8mb coordinator and 2 128gb executors won't work well.

[#5]

To be clear, most production setups with dedicated coordinators that we see given the coordinator the same amount of memory as the executors.

[#6]

@Tim Armstrong  thanks for pointing this out.

We have observed that in our case the memory usage on the coordinator is not that high having co-ordinator of same size as executor will lead to under utilisation of resources on co-ordinator. Or we can have multiple (8) executors of smaller size lets say 32 GB instead of two with 128GB. Please share your thoughts about it