kerberized HDP 2.5 - Beeline not recognizing the kerberos token on node where HiveServer2 is Not installed

[#1]

Hello - i've a Kerberized 8 Node HDP 2.5 cluster... and am facing a peculiar issue.

HiveServer2 is installed on Node-7 & I've a valid Kerberos token for user - hive. When i start beeline on Node-7, i'me able to start it, and am able to query the tables.

--------------------------------------------------------ON NODE-7------------------------------------

[hive@nwk2-bdp-hadoop-07 ~]$ klist Ticket cache: FILE:/tmp/krb5cc_503

Default principal: hive/nwk2-bdp-hadoop-07.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM

Valid starting Expires Service principal 05/04/2017 18:16:45 05/05/2017 04:16:45 krbtgt/AMP.GDCS-QA.APPLE.COM@AMP.GDCS-QA.APPLE.COM renew until 05/11/2017 18:16:45 [hive@nwk2-bdp-hadoop-07 ~]$

beeline -u "jdbc:hive2://nwk2-bdp-hadoop-07.gdcs-qa.apple.com:10000/default;principal=hive/nwk2-bdp-hadoop-07.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM" -n hive

Connecting to jdbc:hive2://nwk2-bdp-hadoop-07.gdcs-qa.apple.com:10000/default;principal=hive/nwk2-bdp-hadoop-07.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37) Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive

--------------------------------------------------------------------------------------------------------------

On Node-6, i've a valid kerberos token, and when i start Beeline, it does not connect and gives GSS initiate error (i.e. token is not being recognized)

Any ideas on what is happening here ?

---------------------------------------------------ON NODE-6 ---------------------------------------

[hive@nwk2-bdp-hadoop-06 ~]$ klist Ticket cache: FILE:/tmp/krb5cc_503

Default principal: hive/nwk2-bdp-hadoop-06.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM

Valid starting Expires Service principal 05/04/2017 22:06:01 05/05/2017 08:06:01 krbtgt/AMP.GDCS-QA.APPLE.COM@AMP.GDCS-QA.APPLE.COM renew until 05/11/2017 22:06:01 [hive@nwk2-bdp-hadoop-06 ~]$

beeline -u "jdbc:hive2://nwk2-bdp-hadoop-07.gdcs-qa.apple.com:10000/;principal=hive/nwk2-bdp-hadoop-06.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM"

Connecting to jdbc:hive2://nwk2-bdp-hadoop-07.gdcs-qa.apple.com:10000/;principal=hive/nwk2-bdp-hadoop-06.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM 17/05/04 22:06:18 [main]: WARN jdbc.HiveConnection: Failed to connect to nwk2-bdp-hadoop-07.gdcs-qa.apple.com:10000 Error: Could not open client transport with JDBC Uri: jdbc:hive2://nwk2-bdp-hadoop-07.gdcs-qa.apple.com:10000/;principal=hive/nwk2-bdp-hadoop-06.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM: Peer indicated failure: GSS initiate failed (state=08S01,code=0) Beeline version 1.2.1000.2.5.3.0-37 by Apache Hive 0: jdbc:hive2://nwk2-bdp-hadoop-07.gdcs-qa.ap (closed)>

[#2]

@Kuldeep Kulkarni, @mqureshi - looping you in , any ideas on this ?

[#3]

@Karan Alang

In your hive-site.xml, what is the value of:

 <name>hive.server2.authentication.kerberos.principal</name>

It should be. See the _HOST. Make sure it is not your host name for node 7.

<value>hive/_HOST@YOUR-REALM.COM</value>

[#4]

@mqureshi

here is the entry in hive-site.xml

<property> <name>hive.server2.authentication.kerberos.principal</name> <value>hive/_HOST@AMP.GDCS-QA.APPLE.COM</value> </property>

i was actually expecting to be able to connect to HiveServer2 (using Beeline) by using the principal specific to the host. eg.

principal=hive/<node7>@AMP.GDCS-QA.APPLE.COM from Node7 and

principal=hive/<node6>@AMP.GDCS-QA.APPLE.COM when connecting from Node6.

is that not correct ?

Do i need to instead use principal=hive/_HOST@AMP.GDCS-QA.APPLE.COM ?

[#5]

@Karan Alang

_HOST is the host where HiveServer2 is running. This is not the client host. Can you please change that to your node 7 but still try to connect from node 6. You'll need the keytab from node 7. If you want to enable access for other users then you'll have to enable impersonation. Following goes into hive-site.xml.

<property>
  <name>hive.server2.enable.impersonation</name>
  <description>Enable user impersonation for HiveServer2</description>
  <value>true</value>
</property>

And then following goes into core-site.xml:

<property>
  <name>hadoop.proxyuser.hive.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.hive.groups</name>
  <value>*</value>
</property

[#6]

hi @Karan Alang,

in the beeline connection string you have to provide the principal of hive user _on_ the Hiveserver, hence it should be the "... 07" even if you connect from node "...06"

Shortcut: try exactly the same beeline command on node 06 as you are executing on node 07

HTH

[#7]

@Karan Alang

With kerberos authentication you need to mention the service principal of hive server2 in beeline connection string. It should of below format.

jdbc:hive2://<host>:<port>/<db>;principal=<Server_Principal_of_HiveServer2>

Refer: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-JDBCClientSe...

If nwk2-bdp-hadoop-07.gdcs-qa.apple.com is your hive server2 then Server_Principal_of_HiveServer2 would be hive/nwk2-bdp-hadoop-07.gdcs-qa.apple.com@AMP.GDCS-QA.APPLE.COM.

This pricipal is same irrespective of client from where you are accessing beeline.

[#8]

How you resolve this issue?i am facing similar..

[#9]

@Karan Alan..Please let me know ASAP. Thanks .