Reply
Expert Contributor
Posts: 64
Registered: ‎11-24-2017

Oozie HiveServer2 credentials with HA enabled and Kerberos

[ Edited ]

Hello everyone, I have High Availabity of HiveServer2 enabled on a kerberized cluster.

I can succesfully connect to beeline with the following command:

 

beeline -u "jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;"

 

My problem is when I try to use Hive2 credentials in Oozie (which afaik uses beeline and jdbc connection as above):

 

<credentials>
        <credential name="hive2_credentials" type="hive2">
            <property>
                <name>hive2.jdbc.url</name>
                <value>jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;</value>
            </property>
            <property>
                <name>hive2.server.principal</name>
                <value>hive/_HOST@AZCLOUD.LOCAL</value>
            </property>
        </credential>
    </credentials>

I took the value of hive2.server.principal fron the hive.server2.authentication.kerberos.principal property in the hive-site.xml, is this correct?

 

This is the hive2 oozie action:

 

<action cred="hive2_credentials" name="HIVE2_ACTION_NODE">
    <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>${package}/hive-site.xml</job-xml>
        <jdbc-url>jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;</jdbc-url>
        <script>${package}/my_query.hql</script>
        <param>nameNode=${nameNode}</param>        
    </hive2>
    <ok to="END_NODE"/>
    <error to="KILL_NODE"/>
</action>

 

I got this error when running the workflow:

 

Connecting to jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 configs from ZooKeeper (state=08S01,code=0)
No current connection
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]

 

Does anyone know how to solve this issue?

 

 

Posts: 1,664
Kudos: 325
Solutions: 262
Registered: ‎07-31-2013

Re: Oozie HiveServer2 credentials with HA enabled and Kerberos

HiveServer2 HA support via ZK is not yet available for supported use in CDH. We do support using a Load Balancer instead: https://www.cloudera.com/documentation/enterprise/latest/topics/admin_ha_hiveserver2.html

If I had to guess what's wrong, without further logging from the Oozie server and action task logs, I'd guess it has something to do with the delegation token support in ZK-based HA mode. Oozie will try to grab the DT for one specific HS2, which other HS2s may not accept if they are not sharing the token secrets via a common store. This is just a theory though, I have no evidence from a test to back this up.

Since this feature has not been tested for wider integration yet in CDH5 (as of CDH 5.14), it is not a supported mode of use.

Would you be able to use the Load Balancer based method instead? This has been tested to work with Oozie and other components.
Highlighted
Expert Contributor
Posts: 64
Registered: ‎11-24-2017

Re: Oozie HiveServer2 credentials with HA enabled and Kerberos

Thank you very much @Harsh J for the detailed answer. I will forward it to the cluster administrators, hoping they will follow the loadbalancer way you suggested ^_^.

Announcements