Support Questions
Find answers, ask questions, and share your expertise

Hive workload management

Hive workload management

Explorer

Running a HDP 3.1 cluster with Hive LLAP enabled. The main problem was working with many concurrent Hive queries. Some of them were very large, occupying the whole cluster, making other concurrent queries waiting for the first one to finish.

 

For this reason I've followed these steps from documentation.

1. Created two resource plans for slow queries and fast queries, based on: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/hive-workload/content/hive_setting_up_and_usin... 

> show resource plans ;

+------------------+---------+--------------------+
| rp_name | status | query_parallelism |
+------------------+---------+--------------------+
| high_concurr_rp | ACTIVE | NULL |
+------------------+---------+--------------------+

 

2. Configured the workflow management, based on this: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/hive-workload/content/hive_interactive_queue.h...

I now have ``hive.server2.tez.interactive.queue=interactive-q`` in custom-site. 

 

My questions are these:

1. How can I check whether those rules from the examples are working? Can I somehow see which pool and what resource allocation is a specific query getting?

2. With the addition of ``hive.server2.tez.interactive.queue``, I've noticed that LLAP is spawning tons of TEZ applications in the ``interactive-q`` queue, so much so that many are not fully running (state=ACCEPTED, FinalStatus=UNDEFINED). How can I control the number (or disable this)? In the ``llap`` queue we have 1 ``llap0`` running as a yarn-service, with 3 supporting TEZ applications. 

tez2.png