Reply
Expert Contributor
Posts: 70
Registered: ‎11-24-2017

Oozie HiveServer2 credentials with HA enabled and Kerberos

[ Edited ]

Hello everyone, I have High Availabity of HiveServer2 enabled on a kerberized cluster.

I can succesfully connect to beeline with the following command:

 

beeline -u "jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;"

 

My problem is when I try to use Hive2 credentials in Oozie (which afaik uses beeline and jdbc connection as above):

 

<credentials>
        <credential name="hive2_credentials" type="hive2">
            <property>
                <name>hive2.jdbc.url</name>
                <value>jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;</value>
            </property>
            <property>
                <name>hive2.server.principal</name>
                <value>hive/_HOST@AZCLOUD.LOCAL</value>
            </property>
        </credential>
    </credentials>

I took the value of hive2.server.principal fron the hive.server2.authentication.kerberos.principal property in the hive-site.xml, is this correct?

 

This is the hive2 oozie action:

 

<action cred="hive2_credentials" name="HIVE2_ACTION_NODE">
    <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>${package}/hive-site.xml</job-xml>
        <jdbc-url>jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;</jdbc-url>
        <script>${package}/my_query.hql</script>
        <param>nameNode=${nameNode}</param>        
    </hive2>
    <ok to="END_NODE"/>
    <error to="KILL_NODE"/>
</action>

 

I got this error when running the workflow:

 

Connecting to jdbc:hive2://trmas-fc2d552a.azcloud.local:2181,trmas-c9471d78.azcloud.local:2181,trmas-6b8bc78c.azcloud.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;ssl=true;
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 configs from ZooKeeper (state=08S01,code=0)
No current connection
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]

 

Does anyone know how to solve this issue?

 

 

Posts: 1,748
Kudos: 365
Solutions: 277
Registered: ‎07-31-2013

Re: Oozie HiveServer2 credentials with HA enabled and Kerberos

HiveServer2 HA support via ZK is not yet available for supported use in CDH. We do support using a Load Balancer instead: https://www.cloudera.com/documentation/enterprise/latest/topics/admin_ha_hiveserver2.html

If I had to guess what's wrong, without further logging from the Oozie server and action task logs, I'd guess it has something to do with the delegation token support in ZK-based HA mode. Oozie will try to grab the DT for one specific HS2, which other HS2s may not accept if they are not sharing the token secrets via a common store. This is just a theory though, I have no evidence from a test to back this up.

Since this feature has not been tested for wider integration yet in CDH5 (as of CDH 5.14), it is not a supported mode of use.

Would you be able to use the Load Balancer based method instead? This has been tested to work with Oozie and other components.
Expert Contributor
Posts: 70
Registered: ‎11-24-2017

Re: Oozie HiveServer2 credentials with HA enabled and Kerberos

Thank you very much @Harsh J for the detailed answer. I will forward it to the cluster administrators, hoping they will follow the loadbalancer way you suggested ^_^.

Announcements