Could you advise if there is a solution to the problem, when Impala assigns heavy query parts to busy executors. For example the following was faced at CDH 5.16 with Impala 2.12.0: Impala has several (let's say 5) executors each having ~100GB RAM. Impala admission control is used. The mem_limit is set default (or about default ~80%), e.g. 80GB. The first relatively long and heavy query (let's name it Query1) comes and one of its steps take ~70GB RAM at executor1, i.e. there is ~10GB available RAM at this executor for reservation. Other 4 executor servers are nearly idle. At the same time the second query (let's name Query2) comes, which requires 40GB RAM, and it might happen the Query2 is assigned to the executor1, which is busy. So the Query2 fails due to it cannot allocate/reserve the memory. Is there a way to configure Impala to assign fragments/query parts to less busy executors? So far the concurrency reduction or reservation removal (since reserved memory amount usually is larger than really used) might work, but I see it too inefficient to use only 1-2 executors out of 5. Impala on YARN potentially might help, but as far as I see, it requires Llama, which is deprecated and is going to be removed soon.
... View more
Greetings. Is it possible to install muitiple, independent Impala services on a single CDH (5.12) environment? We're looking to fully insulate a selection of queries from a large set of ad-hoc queries, and given the current state of memory estimation / memory allocation of Impala in 5.x, separate indepedent Impala services might be the best approach. I was able to install a second instance of Impala, and it immediately complained about port #'s. Refreshing those to +1 for every port # on the second instance solved that problem, but I did start seeing some agents fail (failure to connect to statestore) after a short period of time. Not sure if I'm doomed from the outset here or if there's a path forward. Thanks! M
... View more