Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari Hive view does not reuse YARN(TEZ) application

Ambari Hive view does not reuse YARN(TEZ) application

New Contributor

I use HDP 2.6.3 Virtual Box sandbox with default configuration.

When I connect to hiveserver2 through beeline and execute several queries, all of them are executed in single YARN application (TEZ session).

For example

0: jdbc:hive2://sandbox-hdp.hortonworks.com:2> select count(product_id) from product;           
INFO  : Tez session hasn't been created yet. Opening session                                    
INFO  : Dag name: select count(product_id) from product(Stage-1)                                
INFO  : Status: Running (Executing on YARN cluster with App id application_1516526595231_0008)  
                                                                                                
--------------------------------------------------------------------------------                
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED                
--------------------------------------------------------------------------------                
Map 1 ..........   SUCCEEDED      1          1        0        0       0       0                
Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0                
--------------------------------------------------------------------------------                
VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 3.32 s                      
--------------------------------------------------------------------------------                
+-------+--+                                                                                    
|  _c0  |                                                                                       
+-------+--+                                                                                    
| 1560  |                                                                                       
+-------+--+                                                                                    
1 row selected (7.658 seconds)                                                                  
0: jdbc:hive2://sandbox-hdp.hortonworks.com:2> select count(product_id) from product;           
INFO  : Session is already open                                                                 
INFO  : Dag name: select count(product_id) from product(Stage-1)                                
INFO  : Status: Running (Executing on YARN cluster with App id application_1516526595231_0008)  
                                                                                                
+-------+--+                                                                                    
|  _c0  |                                                                                       
+-------+--+                                                                                    
| 1560  |                                                                                       
+-------+--+                                                                                    
1 row selected (0.9 seconds)                                                                    
0: jdbc:hive2://sandbox-hdp.hortonworks.com:2> select count(product_id) from product;           
INFO  : Session is already open                                                                 
INFO  : Dag name: select count(product_id) from product(Stage-1)                                
INFO  : Status: Running (Executing on YARN cluster with App id application_1516526595231_0008)  
                                                                                                
+-------+--+                                                                                    
|  _c0  |                                                                                       
+-------+--+                                                                                    
| 1560  |                                                                                       
+-------+--+                                                                                    
1 row selected (4.369 seconds)                                                                  

As you can see all three queries used application_1516526595231_0008.

But if I execute queries using Hive View 2.0 it creates a new YARN application for each query. It really slows down performance especially when I'm experimenting and executing a lot of different queries.

I understand that I can set tez.session.am.dag.submit.timeout.secs to some low value so application will not be hanging too long but I do want to make Hive View to reuse existed YARN application as beeline do. Is there any way I can do it?

5 REPLIES 5

Re: Ambari Hive view does not reuse YARN(TEZ) application

New Contributor

@Vitalii Vitrenko Hey there, I'm having the same problem. Did you perhaps find a solution to this problem?

Re: Ambari Hive view does not reuse YARN(TEZ) application

New Contributor

Eventually, I gave up on Hive View and just started to use DbVisualizer which can connect to hive throught jdbc. It works just fine and reuses the same yarn application for different queries.

Highlighted

Re: Ambari Hive view does not reuse YARN(TEZ) application

New Contributor

Did you check the option "tez.am.container.reuse.enabled" is true?

Re: Ambari Hive view does not reuse YARN(TEZ) application

New Contributor

Yes this option is configured true.

Re: Ambari Hive view does not reuse YARN(TEZ) application

Contributor

Hi all,

I had encountered the same situation and I’ve got over that.

In my case, I changed "Start Tez session at Initialization" ("hive.server2.tez.initialize.default.sessions" property) from "false" to "true", and then the problem disappeared.

Don't have an account?
Coming from Hortonworks? Activate your account here