Trying to connect to a kerberized hdp 3.0.1

[#1]

HI,

I am trying to connect to the hive server of my kerberized HDP 3.0.1 using beeline on a external host. I have imported the keytab and also changed the /etc/krb5.conf on the external host, i am able to kinit using the keytab but i stilll get the following error when i am trying to connect to the hive server:

shell> kinit mapr/<FQDN@REALM> -k -t /home/user1/Desktop/hive.service.keytab

shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>"

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/user1/Documents/HDP/apache-hive-3.1.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/user1/Documents/HDP/hadoop-3.0.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Connecting to jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>

Java config name: null

Native config name: /etc/krb5.conf

Loaded from native config

19/07/29 08:49:43 [main]: ERROR transport.TSaslTransport: SASL negotiation failure

javax.security.sasl.SaslException: GSS initiate failed

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_212]

at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[hive-exec-3.1.0.jar:3.1.0]

at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [hive-exec-3.1.0.jar:3.1.0]

at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [hive-exec-3.1.0.jar:3.1.0]

at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:51) [hive-exec-3.1.0.jar:3.1.0]

at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:48) [hive-exec-3.1.0.jar:3.1.0]

at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_212]

at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_212]

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965) [hadoop-common-3.0.1.jar:?]

at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport.open(TUGIAssumingTransport.java:48) [hive-exec-3.1.0.jar:3.1.0]

at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:339) [hive-jdbc-3.1.0.jar:3.1.0]

at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:224) [hive-jdbc-3.1.0.jar:3.1.0]

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.1.0.jar:3.1.0]

at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_212]

at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_212]

at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.Commands.connect(Commands.java:1641) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.Commands.connect(Commands.java:1536) [hive-beeline-3.1.0.jar:3.1.0]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]

at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1384) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1423) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:900) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:795) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1048) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:538) [hive-beeline-3.1.0.jar:3.1.0]

at org.apache.hive.beeline.BeeLine.main(BeeLine.java:520) [hive-beeline-3.1.0.jar:3.1.0]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]

at org.apache.hadoop.util.RunJar.run(RunJar.java:239) [hadoop-common-3.0.1.jar:?]

at org.apache.hadoop.util.RunJar.main(RunJar.java:153) [hadoop-common-3.0.1.jar:?]

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_212]

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_212]

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_212]

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_212]

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_212]

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_212]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_212]

... 36 more

19/07/29 08:49:43 [main]: WARN jdbc.HiveConnection: Failed to connect to <hostname>:10000

Unknown HS2 problem when communicating with Thrift server.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>: GSS initiate failed (state=08S01,code=0)

Beeline version 3.1.0 by Apache Hive

NOTE: I was following the following recommendation, section Using Beeline with Kerberos:

https://mapr.com/docs/archive/mapr40x/Configuring-Hive-on-a-Secure-Cluster_28869090.html#Configuring...

Could you please help?

Thank you

[#2]

Hello @Koffi,

Based on your connection string and problem description, looks like you are not using the right principal in the Beeline connection string:

shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>"

Please use the following:

shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=hive/<FQDN-of-HS2>@REALM>"

Hope this helps!

[#3]

HI Vipin,

THank you for your answer. I still have a error while i try to connect....

shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=hive/<FQDN-of-HS2>@REALM>;auth=kerberos"

...

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_222]

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_222]

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_222]

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_222]

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_222]

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_222]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_222]

... 36 more

19/07/31 11:52:35 [main]: WARN jdbc.HiveConnection: Failed to connect to <hostname>:10000

Unknown HS2 problem when communicating with Thrift server.

Error: Could not open client transport with JDBC Uri: jdbc:hive2://<hostname>:10000/default;principal=hive/<FQDN-of-HS2>@REALM>;auth=kerberos: GSS initiate failed (state=08S01,code=0)

Beeline version 3.1.0 by Apache Hive

NOTE: before running the command i made sure to create the kerberos ticket with a kinit so i am not sure where this no valid credentials provided come from....

[#4]

@Koffi

While logged in as hive can you share the output of

$ klist

Bizarre

[#5]

shell> klist

Ticket cache: FILE:/tmp/krb5cc_1000

Default principal: hive/<FQDN-of-HS2>@REALM

Valid starting Expires Service principal

07/31/2019 10:31:14 08/01/2019 10:31:14 krbtgt/REALM@REALM

I previously copied the /etc/krb5.conf from KDC server to my external host... where i am running my cmd.

[#6]

@Koffi

In your initial posting, you shared this command

shell> kinit mapr/<FQDN@REALM> -k -t /home/user1/Desktop/hive.service.keytab
shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=mapr/<FQDN@REALM>"

Where is the mapr coming from , was this external host part of the cluster before kerberization?

HTH

[#7]

Hi,

It was a typo from my side

but i changed mapr for hive.

[#8]

@Koffi

On your edgenode or node where you copied/imported the keytab ensure you have install the kerberos client software, if it wasn't done do the below steps assuming you are on RHEL or Centos

Remember to revalidate the krb5.conf it should be an exact copy as that on the KDC server.

# yum install -y krb5-libs krb5-workstation

Just for curiosity to be sure what keytab exactly did you copy can you share the name?

Assuming your Kerberos realm is KOFFI.COM and suppose your new host is edgenode.koffi.com. Once the KOFFI.COM realm configuration file (/etc/krb5.conf) has been copied from the KDC to edgenode, use the kadmin protocol set up on the KDC to administer the Kerberos database remotely, directly from edgenode. Add a host principal for our new host and store the host’s secret key in the local keytab file. (kadmin can find the Kerberos admin server from the krb5.conf file you copied.)

edgenode # kadmin -p admin/admin 
Authenticating as principal admin/admin@KOFFI.COM with password. 
Enter password: ******

kadmin:  ank -randkey -policy hosts host/edgenode.koffi.com 
Principal "host/edgenode.koffi.com@KOFFI.COM" created. 

kadmin:  ktadd -k /etc/krb5.keytab host/edgenode.koffi.com 
Entry for principal host/edgenode.koffi.com with kvno 3, encryption type    
Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. 

kadmin:  quit

Now when you run your connection on the beeline CLI

shell> beeline -u "jdbc:hive2://<hostname>:10000/default;principal=hive/<FQDN@KOFFI.COM>"

You shouldn't get "No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)"

Please do that and revert