Support Questions

siraj03 · ‎12-20-2017

I am trying to connect to a Kerberos secured Hiveserver2 that has HA enabled. I am following the approach outlined in this Hive documentation for multi-user scenario (Pre-authenticated subject method) Multi-UserScenarios and Programmatic Login to KerberosKDC

static final String JDBC_DB_URL = "jdbc:hive2://node1:2181,node2:2181,node3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@<realm>;auth=kerberos;kerberosAuthType=fromSubject;";

In the service principal, I have used _HOST since the client program does not know which of the two hiveserver2 nodes is being connected to (HA configuration). According to the documentation

"The _HOST@YOUR-REALM.COM value in the example above is the Kerberos principal for the host where HiveServer2 is running. The special string _HOST in the properties is replaced at run-time by the fully-qualified domain name of the host machine where the daemon is running. This requires that reverse DNS is properly working on all the hosts configured this way. Replace YOUR-REALM.COM with the name of the Kerberos realm your Hadoop cluster is in"

Reverse DNS is configured on all nodes and is verified working. I am able to get hostname from IP as well as IP from hostname. Despite the efforts I could not get this connection to work. It fails with the error

>>>KRBError:
cTime is Sat Oct 22 09:57:54 IST 1988 593497674000
sTime is Thu Dec 14 10:01:16 IST 2017 1513225876000
suSec is 641259 
error code is 7
error Message is Server not found in Kerberos database
cname is username@realm.com 
sname is hive/_host@realm.com 
msgType is 30 
KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER

Can someone please help me resolve this? Thanks!

VR46 · ‎01-29-2018

Hello @Mohamed Ismail Peer,

Can you please try this JDBC command string :

jdbc:hive2://node1:2181,node2:2181,node3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@<realm>

Let us know the Kerberos debug output.

ravindra_punuru · ‎01-29-2018

Hi @Vipin Rathor

Thanks for the reply. The URL which you have specified works fine with kinit step. In our case "kerberosAuthType=fromSubject" is required as we are authenticating multiple users programmatically. BTW, same works fine with Hiveserver2 http mode.

Also, there is a JIRA opened for the same issue

https://issues.apache.org/jira/browse/HIVE-15177

Please review the below details

Below is the URL using:

jdbc:hive2://lvchdp253n1.XXX.com:2181,lvchdp253n2.XXX.com:2181,lvchdp253n3.XXX.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-binary;principal=hive/_HOST@hdp253.XXX.com;auth=kerberos;kerberosAuthType=fromSubject;

Kerberos debug output:

Setting forwardable to true
>>> KeyTabInputStream, readName(): hdp253.XXX.com
>>> KeyTabInputStream, readName(): hdpuser
>>> KeyTab: load() entry length: 77; type: 18
>>> KeyTabInputStream, readName(): hdp253.XXX.com
>>> KeyTabInputStream, readName(): hdpuser
>>> KeyTab: load() entry length: 61; type: 17
>>> KeyTabInputStream, readName(): hdp253.XXX.com
>>> KeyTabInputStream, readName(): hdpuser
>>> KeyTab: load() entry length: 69; type: 16
>>> KeyTabInputStream, readName(): hdp253.XXX.com
>>> KeyTabInputStream, readName(): hdpuser
>>> KeyTab: load() entry length: 61; type: 23
Looking for keys for: hdpuser@hdp253.XXX.com
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
>>> KdcAccessibility: reset
Looking for keys for: hdpuser@hdp253.XXX.com
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
default etypes for default_tkt_enctypes: 16.
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=172.40.9.xxx UDP:88, timeout=30000, number of retries =3, #bytes=151
>>> KDCCommunication: kdc=172.40.9.xxx UDP:88, timeout=30000,Attempt =1, #bytes=151
>>> KrbKdcReq send: #bytes read=684
>>> KdcAccessibility: remove 172.40.9.xxx
Looking for keys for: hdpuser@hdp253.XXX.com
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbAsRep cons in KrbAsReq.getReply hdpuser
logged in successfully.
 Subject:
        Principal: hdpuser@hdp253.XXX.com
        Private Credential: Ticket (hex) = 
0000: 61 82 01 4C 30 82 01 48   A0 03 02 01 05 A1 14 1B  a..L0..H........
0010: 12 68 64 70 32 35 33 2E   64 69 79 6F 74 74 61 2E  .hdp253.XXX.
0020: 63 6F 6D A2 27 30 25 A0   03 02 01 02 A1 1E 30 1C  com.'0%.......0.
0030: 1B 06 6B 72 62 74 67 74   1B 12 68 64 70 32 35 33  ..krbtgt..hdp253
0040: 2E 64 69 79 6F 74 74 61   2E 63 6F 6D A3 82 01 00  .XXX.com....
0050: 30 81 FD A0 03 02 01 12   A1 03 02 01 01 A2 81 F0  0...............
0060: 04 81 ED E8 53 DA 90 76   C7 A6 40 C2 B2 C2 72 0E  ....S..v..@...r.
0070: BA 0A 0A BF 5C E6 17 C5   9F E4 2A EE C5 C4 DF 98  ....\.....*.....
0080: 61 BF F8 63 AD B6 1D E5   AE A7 D5 9B 1C 5E 16 C2  a..c.........^..
0090: 5C 8F 28 C6 9C 42 65 79   CC A0 9A 85 78 D1 97 98  \.(..Bey....x...
00A0: 94 ED 88 4D 60 60 55 4B   BF AB C4 84 F6 72 04 49  ...M``UK.....r.I
00B0: 11 91 E2 A5 C6 B8 15 58   D6 DF ED 63 6C 23 E6 96  .......X...cl#..
00C0: DD 13 22 A5 54 08 51 98   AD F1 47 20 7A 39 A2 82  ..".T.Q...G z9..
00D0: 7A 93 D2 45 68 76 F2 0A   1E 0A 83 FE 76 89 2F 90  z..Ehv......v./.
00E0: DA 67 A1 F5 47 45 3B C7   EC 52 D8 0B 25 D3 58 7C  .g..GE;..R..%.X.
00F0: D8 6D EA 53 8C EA D7 AE   F4 57 35 EF 3C AB 1B B1  .m.S.....W5.<...
0100: 16 52 71 45 0E 4A 1A 53   3F 4F F5 EB 6C 2A C3 12  .RqE.J.S?O..l*..
0110: 26 C8 02 58 BC 46 D0 7D   CF 2F 0E 2F 5B B5 C8 94  &..X.F..././[...
0120: E0 31 2B 01 78 4D 58 36   02 8E A1 03 AA 35 62 FB  .1+.xMX6.....5b.
0130: 5D 40 44 DE 54 8D 8C 21   04 3F 3B 03 EA 74 BA 47  ]@D.T..!.?;..t.G
0140: C4 5B 56 26 55 EB C9 C5   6B 0C 90 F1 98 1D 6F 70  .[V&U...k.....op


Client Principal = hdpuser@hdp253.XXX.com
Server Principal = krbtgt/hdp253.XXX.com@hdp253.XXX.com
Session Key = EncryptionKey: keyType=16 keyBytes (hex dump)=
0000: 51 04 B6 79 BA 94 46 15   10 02 40 73 01 C4 3B 01  Q..y..F...@s..;.
0010: D6 25 2C 02 0E F2 97 3D                            .%,....=




Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time = Tue Jan 30 03:03:55 IST 2018
Start Time = Tue Jan 30 03:03:55 IST 2018
End Time = Wed Jan 31 03:03:55 IST 2018
Renew Till = null
Client Addresses  Null 


Signed on user subject.
 Subject:
        Principal: hdpuser@hdp253.XXX.com
        Private Credential: Ticket (hex) = 
0000: 61 82 01 4C 30 82 01 48   A0 03 02 01 05 A1 14 1B  a..L0..H........
0010: 12 68 64 70 32 35 33 2E   64 69 79 6F 74 74 61 2E  .hdp253.XXX.
0020: 63 6F 6D A2 27 30 25 A0   03 02 01 02 A1 1E 30 1C  com.'0%.......0.
0030: 1B 06 6B 72 62 74 67 74   1B 12 68 64 70 32 35 33  ..krbtgt..hdp253
0040: 2E 64 69 79 6F 74 74 61   2E 63 6F 6D A3 82 01 00  .XXX.com....
0050: 30 81 FD A0 03 02 01 12   A1 03 02 01 01 A2 81 F0  0...............
0060: 04 81 ED E8 53 DA 90 76   C7 A6 40 C2 B2 C2 72 0E  ....S..v..@...r.
0070: BA 0A 0A BF 5C E6 17 C5   9F E4 2A EE C5 C4 DF 98  ....\.....*.....
0080: 61 BF F8 63 AD B6 1D E5   AE A7 D5 9B 1C 5E 16 C2  a..c.........^..
0090: 5C 8F 28 C6 9C 42 65 79   CC A0 9A 85 78 D1 97 98  \.(..Bey....x...
00A0: 94 ED 88 4D 60 60 55 4B   BF AB C4 84 F6 72 04 49  ...M``UK.....r.I
00B0: 11 91 E2 A5 C6 B8 15 58   D6 DF ED 63 6C 23 E6 96  .......X...cl#..
00C0: DD 13 22 A5 54 08 51 98   AD F1 47 20 7A 39 A2 82  ..".T.Q...G z9..
00D0: 7A 93 D2 45 68 76 F2 0A   1E 0A 83 FE 76 89 2F 90  z..Ehv......v./.
00E0: DA 67 A1 F5 47 45 3B C7   EC 52 D8 0B 25 D3 58 7C  .g..GE;..R..%.X.
00F0: D8 6D EA 53 8C EA D7 AE   F4 57 35 EF 3C AB 1B B1  .m.S.....W5.<...
0100: 16 52 71 45 0E 4A 1A 53   3F 4F F5 EB 6C 2A C3 12  .RqE.J.S?O..l*..
0110: 26 C8 02 58 BC 46 D0 7D   CF 2F 0E 2F 5B B5 C8 94  &..X.F..././[...
0120: E0 31 2B 01 78 4D 58 36   02 8E A1 03 AA 35 62 FB  .1+.xMX6.....5b.
0130: 5D 40 44 DE 54 8D 8C 21   04 3F 3B 03 EA 74 BA 47  ]@D.T..!.?;..t.G
0140: C4 5B 56 26 55 EB C9 C5   6B 0C 90 F1 98 1D 6F 70  .[V&U...k.....op


Client Principal = hdpuser@hdp253.XXX.com
Server Principal = krbtgt/hdp253.XXX.com@hdp253.XXX.com
Session Key = EncryptionKey: keyType=16 keyBytes (hex dump)=
0000: 51 04 B6 79 BA 94 46 15   10 02 40 73 01 C4 3B 01  Q..y..F...@s..;.
0010: D6 25 2C 02 0E F2 97 3D                            .%,....=




Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time = Tue Jan 30 03:03:55 IST 2018
Start Time = Tue Jan 30 03:03:55 IST 2018
End Time = Wed Jan 31 03:03:55 IST 2018
Renew Till = null
Client Addresses  Null 


log4j:WARN No appenders could be found for logger (org.apache.hive.jdbc.Utils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Found ticket for hdpuser@hdp253.XXX.com to go to krbtgt/hdp253.XXX.com@hdp253.XXX.com expiring on Wed Jan 31 03:03:55 IST 2018
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
default etypes for default_tgs_enctypes: 16.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=172.40.9.xxx UDP:88, timeout=30000, number of retries =3, #bytes=662
>>> KDCCommunication: kdc=172.40.9.xxx UDP:88, timeout=30000,Attempt =1, #bytes=662
>>> KrbKdcReq send: #bytes read=177
>>> KdcAccessibility: remove 172.40.9.xxx
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
         cTime is Mon Jul 20 10:44:28 IST 2026 1784524468000
         sTime is Tue Jan 30 03:03:57 IST 2018 1517261637000
         suSec is 234162
         error code is 7
         error Message is Server not found in Kerberos database
         cname is hdpuser@hdp253.XXX.com
         sname is hive/_host@hdp253.XXX.com
         msgType is 30
KrbException: Server not found in Kerberos database (7) - UNKNOWN_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hive.service.auth.TSubjectAssumingTransport$1.run(TSubjectAssumingTransport.java:49)
        at org.apache.hive.service.auth.TSubjectAssumingTransport$1.run(TSubjectAssumingTransport.java:46)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hive.service.auth.TSubjectAssumingTransport.open(TSubjectAssumingTransport.java:46)
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:193)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:155)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:270)
        at TestCase_HIVE$2.run(TestCase_HIVE.java:97)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at TestCase_HIVE.getConnection(TestCase_HIVE.java:90)
        at TestCase_HIVE.main(TestCase_HIVE.java:139)
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 27 more
java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: GSS initiate failed
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:217)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:155)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:270)
        at TestCase_HIVE$2.run(TestCase_HIVE.java:97)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at TestCase_HIVE.getConnection(TestCase_HIVE.java:90)
        at TestCase_HIVE.main(TestCase_HIVE.java:139)
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hive.service.auth.TSubjectAssumingTransport$1.run(TSubjectAssumingTransport.java:49)
        at org.apache.hive.service.auth.TSubjectAssumingTransport$1.run(TSubjectAssumingTransport.java:46)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hive.service.auth.TSubjectAssumingTransport.open(TSubjectAssumingTransport.java:46)
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:193)
        ... 9 more
java.lang.NullPointerException
        at TestCase_HIVE.main(TestCase_HIVE.java:140)
Test ended

VR46 · ‎03-28-2018

I stumbled on this today. Not sure if you are still looking for answers but here we go...

Btw, thanks @Ravindra Punuru for debug output.

From the debug output, the Kerberos layer is not able to decipher hive/_host@REALM principal name into correct principal name. Hence the error "Server not found in Kerberos database". Please try replacing the _HOST with FQDN of HiveServer2 node.

Thanks !

Cloudera Community

Support Questions

_HOST in kerberos principal not resolved when connecting to hiveserver2 with HA