Hive Default Queue Won't Work?

[#1]

I have set up my HiveServer2 to use 'hive1,hive2' as its default queues. I expect query submitted by the beeline tool would be running in those two queues, but I observed it runs in the 'default' queue.

I also configured sessions per queue on HS2 to 2, but it seems it does not take effect.

I am using Hive on Tez on HDP 2.3.2 and Ambari 2.1.2.1.

Is there any other configurations required to make the Hive default queues and sessions per queue configs work?

[#2 · Accepted solution]

@yjiang

Add tez.queue.name to your custom hiveserver2-site in Ambari and restart hiveserver2, it will make your default queue.

You can also specify the queue when you connect or before submitting your query:

jdbc:hive2://localhost:10000?tez.queue.name=hive2

or

set tez.queue.name=hive2;

Also check this post from @David Streever for more detailed information:

https://streever.atlassian.net/wiki/pages/viewpage.action?pageId=4390918

[#3]

I followed all the instructions above. But it doesn't work.

Issues I found:

1) After change the session config, Ambari won't restart HS2. It only restarts metastore and webhcat. I had to restart HS2 separately.

2) "Session per queue" config doesn't work. I changed it to 4, and tell Hive to initialize it, but after HS2 restart, it can only start 1 Tez session since the old 1 session per queue config seems to be still effective.

3) Initialized Tez session won't be used. Submitting query from beeline would try to start a new Tez session in the queue but it gets pended again.

[#4]

About #2 I had similar issues with new sessions being stuck with ACCEPTED status in yarn ui for a recently created queue. It happened twice, I'm trying to reproduce it, but seems to be a capacity scheduler issue that you can fix restarting resource manager. After fixing this capacity scheduler issue, default queues and session per queue works fine. Notice that hiveserver2 starts session one-by-one, it may take a while to initialize them all.

About #3, I see the same. I will open a support ticket to get engineering on it. I found few old internal jiras related to this, but not with 2.3.2

[#5]

Thanks, @Guilherme Braccialli

About #2, I am using default RM configs and I can easily re-produce this. Can you please share what kind of CS issue you found and fixed?

About #3, I got your forwarded case email. Please keep me updated.

Thank you!

[#6]

When I created new queue and refreshed RM it only executed one AM on new queue others got stuck in accept state, solved when I restarted RM, but when I repeated same steps it didn't happen again.

[#7]

***** seperating out the thread to test prewarm container ***

@yjiang Change this for prewarm containers

[#8]

Thanks, @Neeraj Sabharwal . I think this prewarm is a little different from the issue I found, which I think is Tez session (AM) initialization and queue config issue. Are they related or depending on?

[#9]

@yjiang Its diff from the original question but then you raised this "it can only start 1 Tez session since the old 1 session per queue config seems to be still effective."

[#10]

Got it. Let me try raising this number.