Struggling to establish jdbc connection with hive on kerborized cluster

[#1]

We have created new java service for establishing connection with hive of Cloudera kerborized cluster, but stuck because of below error. Error remains the same even we generate kerberos ticket for the hive user through its keytab at remote machine. Tried establishing manual connection using the hive beeline utility as well after generating Kerberos ticket but facing the same issue after several hit and trail .

If any one having authentic document or steps to establish jdbc or beeline connection with remote hive for the kerboriszd cluster please share.

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Detail error message is below (with masked host and server deatils>> -


bin]$ ./beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/maxiq/apache-hive-3.0.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/maxiq/hadoop-3.0.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 3.0.0 by Apache Hive
beeline> !connect jdbc:hive2://<<serverFQDN>>:10000/uat;principal=hive/<<server FQDN>>@<<realm>>;ssl=true
Connecting to jdbc:hive2://<<serverFQDN>>:10000/uat;principal=hive/<<server FQDN>>@<<realm>>;ssl=true
25/01/11 23:47:21 [main]: ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_432]
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[hive-exec-3.0.0.jar:3.0.0]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [hive-exec-3.0.0.jar:3.0.0]
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [hive-exec-3.0.0.jar:3.0.0]
at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:51) [hive-exec-3.0.0.jar:3.0.0]
at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:48) [hive-exec-3.0.0.jar:3.0.0]
at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_432]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_432]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.hive.metastore.security.TUGIAssumingTransport.open(TUGIAssumingTransport.java:48) [hive-exec-3.0.0.jar:3.0.0]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:339) [hive-jdbc-3.0.0.jar:3.0.0]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:224) [hive-jdbc-3.0.0.jar:3.0.0]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.0.0.jar:3.0.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_432]
at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_432]
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.Commands.connect(Commands.java:1638) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.Commands.connect(Commands.java:1533) [hive-beeline-3.0.0.jar:3.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_432]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_432]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_432]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_432]
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1362) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1401) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1267) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1067) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:538) [hive-beeline-3.0.0.jar:3.0.0]
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:520) [hive-beeline-3.0.0.jar:3.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_432]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_432]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_432]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_432]
at org.apache.hadoop.util.RunJar.run(RunJar.java:239) [hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:153) [hadoop-common-3.0.0.jar:?]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162) ~[?:1.8.0_432]
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_432]
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189) ~[?:1.8.0_432]
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:218) ~[?:1.8.0_432]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_432]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_432]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_432]
... 35 more
25/01/11 23:47:21 [main]: WARN jdbc.HiveConnection: Failed to connect to <<server FQDN>>:10000
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://<<serverFQDN>>:10000/uat;principal=hive/<<server FQDN>>@<<realm>>;ssl=true: GSS initiate failed (state=08S01,code=0)
beeline>

[#2]

@tosinghkunal, Welcome to our community! To help you get the best possible answer, I have tagged our Hive experts @Asok @smruti @ggangadharan who may be able to assist you further.

Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.

[#3]

Hello @tosinghkunal,

Greetings of the day!

The error message "Failed to find any Kerberos tgt" means there is no Kerberos ticket on the host from where you execute the beeline command.

You can get the Kerberos ticket using kinit command. You can confirm the kerberos ticket using klist command. 

Let us know if this helps.

Cheers!

[#4]

Hi @tj2007

We are able to generate the kerberos ticket using kinit for the hive user's keytab at remote machine and klist showing the valid ticket but still same error message while connecting to hive

[#5]

Hello @tosinghkunal,

Is it the same error message?

Also, I see you are using below connection string:

 !connect jdbc:hive2://<<serverFQDN>>:10000/uat;principal=hive/<<server FQDN>>@<<realm>>;ssl=true

I hope you are using the correct serverFQDN and the realm. If so, can you please share your connection string here?

[#6]

@tosinghkunal ,

Ideally, you don't even need to provide the !connect string it should be automatically connecting to beeline after type beeline.

your cluster is having ssl enabled ? if yes can you check the value of this file - /etc/hive/conf.cloudera.hive_on_tez/beeline-site.xml

it should be similar to below

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->

<configuration>

  <property>

    <name>beeline.hs2.jdbc.url.default</name>

    <value>hive_on_tez</value>

  </property>

  <property>

    <name>beeline.hs2.jdbc.url.hive_on_tez</name>

    <value>jdbc:hive2://ccycloud.cdpy.root.comops.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;trustStoreType=jks;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=sLWNrcWS4SBAqZRtYJ9BVbNxoVwEbfS6hn2stjiezxq</value>

  </property>

</configuration>

[#7]

@tosinghkunal 

When you say remote machine I hope you have the ticket in the machine where you are running the beeline command. 

If you have the valid tickets could you run the beeline command with option --verbose=true and share the output. Be sure to mask the hostnames when you share the file. 

[#8]

@tosinghkunal, Did any of the responses assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.

[#9 · Accepted solution]

Hi All,

It was issue related to config setting. After setting the environment properly issue got resolved. It seems due to some misconfiguration of JAAS and krb5.conf. Following below blog has resolved the issue

http://shantanu01.blogspot.com/2020/06/how-to-setup-beeline-on-linux-for.html