About dennis_wang

dennis_wang · ‎01-26-2022

Hi, I'm not sure which situation you are indicating. Your query might been restricted by MEM_LIMIT. process limit: set by --mem-limit startup option query limit: set by MEM_LIMIT in your query through impala-shell or jdbc... plz check this reference: https://impala.apache.org/docs/build/html/topics/impala_mem_limit.html

dennis_wang · ‎01-26-2022

Is memory distributed on average across all nodes during daemon query execution? https://impala.apache.org/docs/build/html/topics/impala_admission.html In "Clamp MEM_LIMIT Query Option" section, IF "Maximum Query Memory Limit" for the pool set to 10GB, then the query can use 50GB for the calculation is (Maximum Query Memory Limit * number of Impala nodes). eg. Query estimates for 2.5GB per-host , even there's enough memory left in other nodes, query will failed IF only 1 node out of 2.5GB.

dennis_wang · ‎01-25-2022

Hi guys, Is there a way to isolate impala daemon resources in one cluster? eg. 6 daemons in 1 Impala Cluster. Is there a method to isolate 2 daemons' resources for testing, 4 daemons' resources for production? Testing queries only use the memory resource in their daemons. HAProxy can forward requests to different daemons, but memory resources are not isolated. If query from testing takes too much memories, production query may failed while executing. The only way I can figure out is setup another impala cluster. I know impala provides admission control, what if the query execution plan estimate memory cost wrong, the actual memory cost is far more than the estimation(eg. 3G VS 150G)? What parameters limit the actual memory usage?

Online	Offline
Last Visited	‎02-11-2022 03:54 AM

Member Since	‎01-14-2019 12:58 PM
Last Visited	‎02-11-2022 03:54 AM
Posts	4

Cloudera Community

Re: Impala Memory limit exceed

Re: Impala Memory limit exceed

Re: Impala resource Isolation

Impala resource Isolation