Hi, thanks for the clarification. I am working on CDH 5.13+. Maybe I was not exact, but my experience is that Impala very often cant run the query because it requires more memory than it can get. It is true that query in running state fails rarely on "memory limit" if the pools are set to a non-overlapping mode (i.e. total memory of Impala daemons divided for example by 3 pools - such as M, L, XL, and in each pool you allow to run only that much queries how much do you have in the pool - such as XL has a 9GB limit, the pool size is 27GB, you allow 3 queries at a time to run). But my complain was that in general, that many times the query (with up-to-date stats on tables) simply does not start to run in the corresponding pool (for example pool L with 2GB limit) because the planner thinks it will need more memory. And this is what I was referring to, that you as and end user can't trade this for a disk spill, cant say, hey I know that maybe the sum of the operators are above 2GB in your plan, but please run it in this pool with 2G limit, and if you will hit the wall, start to spill (i.e. trade off for performance). So my experience is, that the user has to "upgrade" the query to pool XL, and run it there. But then it is like in a "highway" - everybody is driving on the left lane (or right - for UK readers). And the funny thing is that after the query finishes in XL pool, you examine the memory allocations and it would fit into the 2G limit. And @Tim Armstrong thank you very much for all the efforts, I have a quite experience and I know the history of Impala, so thumbs up for all the improvements done in the last versions - looking forward to CDH6!
... View more
