Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

configure hive.auto.convert.join.noconditionaltask.size via Ambari is not visible in beeline client?

avatar
Contributor

screenshot-from-2017-10-11-11-55-06.png

Hi,

Parameters configured via Ambari is not applied. Why?

Via Amabari It was configured hive.auto.convert.join.noconditionaltask.size=2600000

Connected to: Apache Hive (version 2.1.0.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
0: jdbc:hive2://ip-172-31-35-100.us-west-2.co> set hive.auto.convert.join.noconditionaltask.size;
+----------------------------------------------------------+--+
|                           set                            |
+----------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=858783744  |
+----------------------------------------------------------+--+
    <property>
      <name>hive.auto.convert.join.noconditionaltask</name>
      <value>true</value>
    </property>
   
    <property>
      <name>hive.auto.convert.join.noconditionaltask.size</name>
      <value>858783744</value>
    </property>
   

Thank you,

1 ACCEPTED SOLUTION

avatar
Super Guru

@Yevgen Shramko,

For LLAP server , this should be configured under Advanced hive-interactive site.

Check screenshot attached

Thanks,

Aditya

40789-screen-shot-2017-10-11-at-53426-pm.png

View solution in original post

5 REPLIES 5

avatar
Master Mentor

@Yevgen Shramko

I tried the same on HDP 2.6.1 (Ambari 2.5.1) and i can see that the changes are getting reflected after making the changes via Ambari When we do "Restart All Required" services.

Example:

Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://amb25102.example.com:2181,amb> set hive.auto.convert.join.noconditionaltask.size;
+----------------------------------------------------------+--+
|                           set                            |
+----------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=333333333  |
+----------------------------------------------------------+--+
1 row selected (0.302 seconds)
1: jdbc:hive2://amb25102.example.com:2181,amb> 

.

39773-beeline-issue.png

.

Please check if the client configs are reflected properly or not? And the host which is running the beeline is synced with Ambari (means ambari agent is running fine on that host and able to communication with Ambari)

- Also most important "Restart All Required" is executed".

avatar
Master Mentor

@Yevgen Shramko

Also please check the if you are looking at the right file:

[root@amb25103 etc]#  find /etc -name "hive-site.xml" | xargs grep -A 3 -i "hive.auto.convert.join.noconditionaltask.size"
/etc/hive/conf.backup/hive-site.xml:      <name>hive.auto.convert.join.noconditionaltask.size</name>
/etc/hive/conf.backup/hive-site.xml-      <value>286331153</value>
/etc/hive/conf.backup/hive-site.xml-    </property>
/etc/hive/conf.backup/hive-site.xml-    
--
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml:      <name>hive.auto.convert.join.noconditionaltask.size</name>
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml-      <value>333333333</value>
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml-    </property>
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml-    
--
/etc/hive/2.6.1.0-129/0/hive-site.xml:      <name>hive.auto.convert.join.noconditionaltask.size</name>
/etc/hive/2.6.1.0-129/0/hive-site.xml-      <value>333333333</value>
/etc/hive/2.6.1.0-129/0/hive-site.xml-    </property>
/etc/hive/2.6.1.0-129/0/hive-site.xml-    

<br>

.

And the Symlinks are correct:

[root@amb25103 etc]# ls -l /etc/hive
total 4
drwxr-xr-x. 3 root root   14 Jun  9 11:16 2.6.1.0-129
lrwxrwxrwx. 1 root root   33 Jun  9 11:16 conf -> /usr/hdp/current/hive-client/conf
drwxr-xr-x. 3 root root 4096 Jun  9 11:16 conf.backup

[root@amb25103 etc]# ls -l /usr/hdp/current/hive-client/conf
lrwxrwxrwx. 1 root root 23 Jun  9 11:16 /usr/hdp/current/hive-client/conf -> /etc/hive/2.6.1.0-129/0

.

avatar
Contributor

@Jay SenSharma

Ok. It looks strange as for me.

1. The one is Correct.

$ beeline -u "jdbc:hive2://ip-172-31-35-100.us-west-2.compute.internal:2181,ip-172-31-34-50.us-west-2.compute.internal:2181,ip-172-31-34-136.us-west-2.compute.internal:2181/bench_mtu_p;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" -n hive -p admin

Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
0: jdbc:hive2://ip-172-31-35-100.us-west-2.co> set hive.auto.convert.join.noconditionaltask.size;
+--------------------------------------------------------+--+
|                          set                           |
+--------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=2600000  |
+--------------------------------------------------------+--+ 

2. But I would like to change here. As I understood this is LLAP server

$ beeline -u "jdbc:hive2://ip-172-31-35-100.us-west-2.compute.internal:2181,ip-172-31-34-50.us-west-2.compute.internal:2181,ip-172-31-34-136.us-west-2.compute.internal:2181/bench_mtu_p;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2" -n hive -p admin

Connected to: Apache Hive (version 2.1.0.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
0: jdbc:hive2://ip-172-31-35-100.us-west-2.co> set hive.auto.convert.join.noconditionaltask.size;
+----------------------------------------------------------+--+
|                           set                            |
+----------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=858783744  |
+----------------------------------------------------------+--+

avatar
Super Guru

@Yevgen Shramko,

For LLAP server , this should be configured under Advanced hive-interactive site.

Check screenshot attached

Thanks,

Aditya

40789-screen-shot-2017-10-11-at-53426-pm.png

avatar
Contributor

@Aditya Sirna That's it. Thank you so much.