impala memory limit exceed

[#1]

Hi,

We have 5 nodes and impala mem limit (Impala Daemon Memory Limit) is 22G, when we run two queries, we had memory limit exceed error, when look at the usage

Query(5a48ee040d09acb9:f9d5060000000000): Total=2.76 GB Peak=3.80 GB
  Fragment 5a48ee040d09acb9:f9d5060000000000: Total=233.00 B Peak=280.00 KB
    EXCHANGE_NODE (id=9): Total=0 Peak=0
    CodeGen: Total=233.00 B Peak=272.00 KB
  Block Manager: Limit=17.60 GB Total=2.76 GB Peak=3.70 GB

Query(93418d547f9cc01e:ea4f055300000000): Total=0 Peak=2.07 GB

no other queries are running, do we miss something?

Thanks

Shannon

[#2]

I'm not sure what is going on.  You are right that 2.07G + 3.80G should fit in 22G.

It's possible that some queries haven't fully closed and released their resources. You could look in CM to see what queries are running, or look in the /memz page of the Impalad debug webpages to see what queries are consuming memory on each daemon.