Could you advise if there is a solution to the problem, when Impala assigns heavy query parts to busy executors. For example the following was faced at CDH 5.16 with Impala 2.12.0:   Impala has several (let's say 5) executors each having ~100GB RAM. Impala admission control is used. The mem_limit is set default (or about default ~80%), e.g. 80GB. The first relatively long and heavy query (let's name it Query1) comes and one of its steps take ~70GB RAM at executor1, i.e. there is ~10GB available RAM at this executor for reservation. Other 4 executor servers are nearly idle. At the same time the second query (let's name Query2) comes, which requires 40GB RAM, and it might happen the Query2 is assigned to the executor1, which is busy. So the Query2 fails due to it cannot allocate/reserve the memory.   Is there a way to configure Impala to assign fragments/query parts to less busy executors? So far the concurrency reduction or reservation removal (since reserved memory amount usually is larger than really used) might work, but I see it too inefficient to use only 1-2 executors out of 5. Impala on YARN potentially might help, but as far as I see, it requires Llama, which is deprecated and is going to be removed soon. ... View more

Thank you very much Tim for providing this insight.  I have assumption that MEM_LIMIT option is asking for that amount of space for query. ... View more

Do you have any information when 5.15.2/5.16.2 version will be released? Thank you ... View more

@Andreyeff Another thing you can try doing is increasing the raft heartbeat interval from 500ms to 1500ms or even 3000ms, see https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_raft_heartbeat_interval_ms   This will affect your recovery time by a few seconds if a leader fails since by default, elections don't happen for 3 missed heartbeat periods (controlled by https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_leader_failure_max_missed_heartbeat_periods ) ... View more

Right, to make the fix into Impala it's necessary to relink impalad with patched Kudu client.   impalad is linked against the kudu_client dynamically, so in theory it might be possible just to replace the libkudu_client.so.0 library with the patched version.  However, that's really messy and I would not recommend that.  If you use CDH anyway, the best option is to wait for next release of CDH.  I don't know what version that will be, though.   If you want a workaround, set the -- authn_token_validity_seconds flag to be months or even one year long  (i.e.  -- authn_token_validity_seconds= 31536000) and restart Kudu masters.  You will need to enable experimental flags as well (i.e. add  --unlock_experimental_flags). ... View more

Oh, the issue appeared to be even in functions. Thanks for fast reply and raising a ticket. ... View more

Hi Andreyeff, it's not recommended to run past the scale limits because it's not tested. Likely you will run into scalability problems due to excessive numbers of threads and context switching, and other things we have not explored yet. ... View more