Our client is running impala query and he see something like this
Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
Upon checking the logs I dont see it's causing any impact or issues but trying to understand what does this mean and why did we see this message
Can you let anyone let me know if you had came across this and what actions were taken ..
Check the Impala Daemon Mem Limit from Cloudera Manager. What is it set at?
I have noticed that, if the MEM_LIMIT for Daemon is 8GB. And if the per-host mem cost is larger than the set 8GB, I get a similar memory error message.
The ideal solution is to bump up the Mem Limit per Daemon, and if you have Yarn Admission Control enabled, you will need to adjust those values too.
I strongly recommend, to tune YARN memory configurations first, before changing anything else. This helped solve alot of problems at our end.
Second, if you are using YARN Admision Control - then ideal Default Query Memory Limit and set it. The smaller the number, more concurrent queries.