question Re: HIVE LLAP: managing concurrent queries in Archives of Support Questions (Read Only)

HIVE LLAP: managing concurrent queries

mRabramS — Mon, 27 Aug 2018 19:09:39 GMT

When running a large Hive query that takes up the whole cluster (all LLAP executors are busy), simultaneously trying to run even a simplest query takes a lot of time, probably because the process is waiting for a free executor to work with.

Is it possible to restrict/manage the queues in a similar manner as does YARN, i.e., so that there would always be at least one free executor for a quick query, or that the tasks from concurrent queries would be pushed in front of the line to guarantee some minimal amount of exposure to the executors these tasks would get?

Re: HIVE LLAP: managing concurrent queries

sergey — Tue, 28 Aug 2018 02:02:34 GMT

The functionality like that is available in Apache Hive 3.0 (bugfixes in 3.1), as well as HDP 3.0/1; see the "workload management" feature. https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/hive-workload/content/hive_workload_management.html and https://issues.apache.org/jira/browse/HIVE-17481

Re: HIVE LLAP: managing concurrent queries

mRabramS — Tue, 28 Aug 2018 02:05:38 GMT

Oh, that's good to hear, looking forward to upgrade then! Thank you for the heads up!