Getting an error when connecting to Hive with Kerberos

[#1]

I get the following error when connecting to Hive.

Exception in thread "main" java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: Invalid status 21.
	at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
	at com.cloudera.hiveserver2.hivecommon.api.ServiceDiscoveryFactory.createClient(Unknown Source)
	at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)

My JDBC connection string is

jdbc:hive2://<FQDNHost>:10000/default;AuthMech=1;KrbAuthType=1;KrbHostFQDN=<FQDNHost>;KrbRealm=<MYREAL>;KrbServiceName=hive;LogLevel=6;LogPath=d:/TestPLPFolder/hivejdbclog

The hivejdbclog doesn't really get me any good information. 

Does anybody have any idea what the Invalid Status 21 means?

[#2]

 The error usually happens when you try to connect to ssl enabled hs2 with plaintext connection. 

a.Which version of CDH/HDP are you using?

b. Can you check in HS2 logs exactly during the timestamp the error "Unable to connect to server: Invalid status 21" was reported on client. The error you notice on server side will give further clues

c. Do you have SSL enabled on HS2 ?

[#3]

a) Hortonworks Hive 3.1.0.3.1.0.0-78, but using Cloudera JDBC driver.2.6.9.1011.

b) Here's the message on the server when I try to connect (/var/log/hive/hiveserver2.log)

2020-03-10T15:41:41,954 ERROR [HiveServer2-Handler-Pool: Thread-185]: server.TThreadPoolServer (:()) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
        at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_112]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
        at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        ... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) ~[?:1.8.0_112]
        at sun.security.ssl.InputRecord.read(InputRecord.java:527) ~[?:1.8.0_112]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[?:1.8.0_112]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:1.8.0_112]
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) ~[?:1.8.0_112]
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[?:1.8.0_112]
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_112]
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_112]
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_112]
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
        ... 10 more

c) I believe so.  When I try to add an SSL flag

jdbc:hive2://vdbdgw01dsy.dsone.3ds.com:10000/default;AuthMech=1;KrbAuthType=1;KrbHostFQDN=vdbdgw01dsy.dsone.3ds.com;KrbRealm=DSONE.3DS.COM;KrbServiceName=hive;LogLevel=6;LogPath=d:/TestPLPFolder/hivejdbclog;SSL=1;

though I get the following

Exception in thread "main" java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
	at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
	at com.cloudera.hiveserver2.hivecommon.api.ServiceDiscoveryFactory.createClient(Unknown Source)
	at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
	at HiveJDBCClientExample.connectViaDM(HiveJDBCClientExample.java:32)
Caused by: com.cloudera.hiveserver2.support.exceptions.GeneralException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
	... 9 more
Caused by: java.lang.RuntimeException: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.cloudera.hiveserver2.hivecommon.api.HiveServerPrivilegedAction.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:361)
	at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
	at com.cloudera.hiveserver2.hivecommon.api.ServiceDiscoveryFactory.createClient(Unknown Source)
	at com.cloudera.hiveserver2.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
	at com.cloudera.hiveserver2.jdbc.common.AbstractDriver.connect(Unknown Source)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
	at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
	at HiveJDBCClientExample.connectViaDM(HiveJDBCClientExample.java:32)
	at HiveJDBCClientExample.main(HiveJDBCClientExample.java:48)
Caused by: com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
	at com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TSaslTransport.sendSaslMessage(TSaslTransport.java:166)
	at com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:106)
	at com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
	at com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38)
	... 13 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:641)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:460)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
	at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:970)
	at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
	at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
	at com.cloudera.hive.jdbc42.internal.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
	... 17 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:233)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:110)
	at com.cloudera.hiveserver2.jdbc.common.DSTrustManager.checkServerTrusted(Unknown Source)
	at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1510)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625)
	... 32 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
	... 40 more

[#4 · Accepted solution]

This is much clear now

On server side the request was rejected as the client was initiating non-ssl connection

Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

Client side it was unable to  trust the server certs as it was not configured to use a truststore

Caused by: com.cloudera.hiveserver2.support.exceptions.GeneralException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][HiveJDBCDriver](500169) Unable to connect to server: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

You got to add few more properties to your connection string

jdbc:hive2://vdbdgw01dsy.dsone.3ds.com:10000/default;AuthMech=1;KrbAuthType=1;KrbHostFQDN=vdbdgw01dsy.dsone.3ds.com;KrbRealm=DSONE.3DS.COM;KrbServiceName=hive;LogLevel=6;LogPath=d:/TestPLPFolder/hivejdbclog;SSL=1;SSLTrustStore=<path_to_truststore>;SSLTrustStorePwd=<password to truststore>
If you dont have password to your truststore you can omit the parameter SSLTrustStorePwd