question Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design in Support Questions

Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design

wfloyd — Mon, 19 Aug 2019 12:57:17 GMT

We've been using the Tez Persistent Queues (Interactive Query Tuning) to optimize Tez Queue performance. Separately we now have Capacity Scheduler "Fair Sharing" policies, which allows separate jobs on the same queue to execute with evenly shared resources.

How should we reconcile the two for optimal Hive configuration? E.g. should we configure Persistent queues when queries are submitted as one user "Hive" and use Fair Sharing policies if the job is submitted by a variety of users?

Any guidance on whether or not we should use the two in combination will help.

Re: Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design

nsabharwal — Tue, 20 Oct 2015 18:18:10 GMT

Should we configure Persistent queues when queries are submitted as one user "Hive" and use Fair Sharing policies if the job is submitted by a variety of users?

Hi Wes,

This makes sense. Looking at the docs, if there are multiple users (users like u1, u2 , u3 ) hitting the queue then fair share will help "if there is a query running already in a queue and taking up all of the resources, when the second session with a query is introduced, the sessions eventually end up with equal numbers of resources per session. Initially, there is a delay, but if ten queries are run concurrently most of the time, the resources are divided equally among them."

Are you looking for benchmarks or performance numbers for this "fair share vs. non"?

Re: Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design

wfloyd — Tue, 20 Oct 2015 23:16:23 GMT

No need for benchmarks or performance numbers. Between using "fair share" and "tez persistent queues" , I'm curious if we should use both techniques in tandem OR understand when we should choose one vs the other?

Perhaps the "fair share" approach is best when trying to reconcile many users sharing resources, then "tez persistent queues" are valuable when absolute lowest latency for queries is the primary goal?

Re: Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design

aervits — Thu, 04 Feb 2016 00:10:11 GMT

@Wes Floyd are you still having issues with this? Can you accept best answer or provide your own solution?

Re: Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design

wfloyd — Thu, 04 Feb 2016 04:13:13 GMT

@Artem Ervits - this question has not yet been completely answered.

Re: Hive User Concurrency - Reconciling YARN Capacity Scheduler Fair Sharing Policy and Tez Persistent Queue Design

aervits — Thu, 04 Feb 2016 10:54:50 GMT

@Wes Floyd

I pinged sme-hive for an answer, @gopal responded with the following statement. Half of all interactive tuning will be replaced by LLAP. Hive 2.0 is days from being released in the Apache.