configure hive.auto.convert.join.noconditionaltask.size via Ambari is not visible in beeline client?

[#1]

screenshot-from-2017-10-11-11-55-06.png

Hi,

Parameters configured via Ambari is not applied. Why?

Via Amabari It was configured hive.auto.convert.join.noconditionaltask.size=2600000

Connected to: Apache Hive (version 2.1.0.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
0: jdbc:hive2://ip-172-31-35-100.us-west-2.co> set hive.auto.convert.join.noconditionaltask.size;
+----------------------------------------------------------+--+
|                           set                            |
+----------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=858783744  |
+----------------------------------------------------------+--+

    <property>
      <name>hive.auto.convert.join.noconditionaltask</name>
      <value>true</value>
    </property>
   
    <property>
      <name>hive.auto.convert.join.noconditionaltask.size</name>
      <value>858783744</value>
    </property>
   

Thank you,

[#2]

@Yevgen Shramko

I tried the same on HDP 2.6.1 (Ambari 2.5.1) and i can see that the changes are getting reflected after making the changes via Ambari When we do "Restart All Required" services.

Example:

Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://amb25102.example.com:2181,amb> set hive.auto.convert.join.noconditionaltask.size;
+----------------------------------------------------------+--+
|                           set                            |
+----------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=333333333  |
+----------------------------------------------------------+--+
1 row selected (0.302 seconds)
1: jdbc:hive2://amb25102.example.com:2181,amb> 

.

.

Please check if the client configs are reflected properly or not? And the host which is running the beeline is synced with Ambari (means ambari agent is running fine on that host and able to communication with Ambari)

- Also most important "Restart All Required" is executed".

[#3]

@Yevgen Shramko

Also please check the if you are looking at the right file:

[root@amb25103 etc]#  find /etc -name "hive-site.xml" | xargs grep -A 3 -i "hive.auto.convert.join.noconditionaltask.size"
/etc/hive/conf.backup/hive-site.xml:      <name>hive.auto.convert.join.noconditionaltask.size</name>
/etc/hive/conf.backup/hive-site.xml-      <value>286331153</value>
/etc/hive/conf.backup/hive-site.xml-    </property>
/etc/hive/conf.backup/hive-site.xml-    
--
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml:      <name>hive.auto.convert.join.noconditionaltask.size</name>
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml-      <value>333333333</value>
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml-    </property>
/etc/hive/2.6.1.0-129/0/conf.server/hive-site.xml-    
--
/etc/hive/2.6.1.0-129/0/hive-site.xml:      <name>hive.auto.convert.join.noconditionaltask.size</name>
/etc/hive/2.6.1.0-129/0/hive-site.xml-      <value>333333333</value>
/etc/hive/2.6.1.0-129/0/hive-site.xml-    </property>
/etc/hive/2.6.1.0-129/0/hive-site.xml-    

<br>

.

And the Symlinks are correct:

[root@amb25103 etc]# ls -l /etc/hive
total 4
drwxr-xr-x. 3 root root   14 Jun  9 11:16 2.6.1.0-129
lrwxrwxrwx. 1 root root   33 Jun  9 11:16 conf -> /usr/hdp/current/hive-client/conf
drwxr-xr-x. 3 root root 4096 Jun  9 11:16 conf.backup

[root@amb25103 etc]# ls -l /usr/hdp/current/hive-client/conf
lrwxrwxrwx. 1 root root 23 Jun  9 11:16 /usr/hdp/current/hive-client/conf -> /etc/hive/2.6.1.0-129/0

.

[#4]

@Jay SenSharma

Ok. It looks strange as for me.

1. The one is Correct.

$ beeline -u "jdbc:hive2://ip-172-31-35-100.us-west-2.compute.internal:2181,ip-172-31-34-50.us-west-2.compute.internal:2181,ip-172-31-34-136.us-west-2.compute.internal:2181/bench_mtu_p;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" -n hive -p admin

Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
0: jdbc:hive2://ip-172-31-35-100.us-west-2.co> set hive.auto.convert.join.noconditionaltask.size;
+--------------------------------------------------------+--+
|                          set                           |
+--------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=2600000  |
+--------------------------------------------------------+--+ 

2. But I would like to change here. As I understood this is LLAP server

$ beeline -u "jdbc:hive2://ip-172-31-35-100.us-west-2.compute.internal:2181,ip-172-31-34-50.us-west-2.compute.internal:2181,ip-172-31-34-136.us-west-2.compute.internal:2181/bench_mtu_p;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2" -n hive -p admin

Connected to: Apache Hive (version 2.1.0.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
0: jdbc:hive2://ip-172-31-35-100.us-west-2.co> set hive.auto.convert.join.noconditionaltask.size;
+----------------------------------------------------------+--+
|                           set                            |
+----------------------------------------------------------+--+
| hive.auto.convert.join.noconditionaltask.size=858783744  |
+----------------------------------------------------------+--+

[#5 · Accepted solution]

@Yevgen Shramko,

For LLAP server , this should be configured under Advanced hive-interactive site.

Check screenshot attached

Thanks,

Aditya

[#6]

@Aditya Sirna That's it. Thank you so much.