Following is our Impala cluster config
As most of the query processing happens on the executor we have kept the executor's size more and co-ordinator's size small.
Now when we enable Impala admission controls we run into the situation where queries having memory requirements more than that of coordinator's memory gets rejected because of insufficient memory error, even though the cluster on the whole has enough memory to process the request. This is the error message which I get
Rejected query from pool root.default: request memory needed 8.29 GB per node is greater than memory available for admission 6.00 GB of <coordinator's DNS>. Use the MEM_LIMIT query option to indicate how much memory is required per node.
Following is the admission control config
As of now we only one default pool
Max Memory :- 200GB
Default Memory Limit :- No default limit
Max queued queries :- 200
Queue Timeout :- 5 Minutes
What can be the reasons for this error? Does executor's and coordinator's hardware size need to be same? Can I exclude coordinator from the admission control?
In Impala in CDH6, queries reserve the same amount of memory on the coordinator as on the executors. I.e. coordinators need to be given enough memory so that queries can reserve the same amount on the coordinator as executors. So the config with 1 8mb coordinator and 2 128gb executors won't work well.
@Tim Armstrong thanks for pointing this out.
We have observed that in our case the memory usage on the coordinator is not that high having co-ordinator of same size as executor will lead to under utilisation of resources on co-ordinator. Or we can have multiple (8) executors of smaller size lets say 32 GB instead of two with 128GB. Please share your thoughts about it