Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive beeline Connection string for Wire encryption SSL with Kerberos not working

Highlighted

Hive beeline Connection string for Wire encryption SSL with Kerberos not working

New Contributor

I am using HDP 2.6.5, HS2 is kerberized & HA enabled. remote ZK connection string is not working but locally Kerberos connection string works

Got stuck between Kerberos HS2 HA & SSL wire encryption for Hive. Need your help,
1) Before enabling SSL, Hive with Kerberos was working from local hiveserver2 beeline

[hive@server1 hive]$ beeline
Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive
beeline> !connect jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Connecting to jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Enter username for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
Enter password for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
Connected to: Apache Hive (version 1.2.1000.2.6.5.0-292)
Driver: Hive JDBC (version 1.2.1000.2.6.5.0-292)
Transaction isolation: TRANSACTION_REPEATABLE_READ

post SSL enabling:-

beeline> !connect jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Connecting to jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Enter username for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
Enter password for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
18/10/23 06:43:05 [main]: WARN jdbc.HiveConnection: Failed to connect to server1-IP:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM: Invalid status 21 (state=08S01,code=0)
0: jdbc:hive2://server1-IP:10000/default (closed)>


Note - However the zookeeper connection string in HA mode establishes the connection and again it disconnects and was not working from remote server, see below error
I tried changing the transport mode from binary to http but that did not worked and reverted to binary.

beeline> !connect jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
Connecting to jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
Enter username for jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;:
Enter password for jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;:
2018-10-23 05:46:22 INFO Utils:310 - Supplied authorities: ZK1:2181,ZK2:2181,ZK3:2181
2018-10-23 05:46:22 INFO CuratorFrameworkImpl:224 - Starting
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:host.name=Remore-Server
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.version=1.8.0_181
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.class.path=/usr/lib/python2.7/site-packages/pyspark/conf:/usr/lib/python2.7/site-packages/pyspark/jars/JavaEWAH-0.3.2.jar:/usr/lib/python2.7/site-packages/pyspark/jars/RoaringBitmap-0.5.11.jar:/usr/lib/python2.7/site-packages/pyspark/jars/ST4-4.0.4.jar:/usr/lib/python2.7/site-packages/pyspark/jars/activation-1.1.1.jar:/usr/lib/python2.7/site-packages/pyspark/jars/aircompressor-0.8.jar:/usr/lib/python2.7/site-packages/pyspark/jars/antlr-2.7.7.jar:/usr/lib/python2.7/site-packages/pyspark/jars/antlr-runtime-3.4.jar:/usr/lib/python2.7/site-packages/pyspark/jars/antlr4-runtime-4.7.jar:/usr/lib/python2.7/site-packages/pyspark/jars/aopalliance-1.0.jar:/usr/lib/python2.7/site-packages/pyspark/jars/aopalliance-repackaged-2.4.0-b34.jar:/usr/lib/python2.7/site-packages/pyspark/jars/apache-log4j-extras-1.2.17.jar:/usr/lib/python2.7/site-packages/pyspark/jars/apacheds-i18n-2.0.0-M15.jar:/usr/lib/python2.7/site-packages/pyspark/jars/apa............
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.compiler=<NA>
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:os.name=Linux
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:os.arch=amd64
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:os.version=3.10.0-693.el7.x86_64
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:user.name=hive
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:user.home=/home/hive
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:user.dir=/home/hive
2018-10-23 05:46:22 INFO ZooKeeper:438 - Initiating client connection, connectString=ZK1:2181,ZK2:2181,ZK3:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@3e77a1ed
2018-10-23 05:46:22 INFO ClientCnxn:975 - Opening socket connection to server ZK1/ZK1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-23 05:46:22 INFO ClientCnxn:852 - Socket connection established to ZK1/ZK1:2181, initiating session
2018-10-23 05:46:22 INFO ClientCnxn:1235 - Session establishment complete on server ZK1/ZK1:2181, sessionid = 0x1667e7f79ba0086, negotiated timeout = 60000
2018-10-23 05:46:22 INFO ConnectionStateManager:228 - State change: CONNECTED
2018-10-23 05:46:23 INFO ZooKeeperHiveClientHelper:83 - Selected HiveServer2 instance with uri: hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM
2018-10-23 05:46:23 INFO ZooKeeper:684 - Session: 0x1667e7f79ba0086 closed
2018-10-23 05:46:23 INFO ClientCnxn:512 - EventThread shut down
2018-10-23 05:46:23 INFO Utils:397 - Resolved authority: hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM
2018-10-23 05:46:23 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO HiveConnection:208 - Could not open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO CuratorFrameworkImpl:224 - Starting
2018-10-23 05:46:23 INFO ZooKeeper:438 - Initiating client connection, connectString=ZK1:2181,ZK2:2181,ZK3:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@7995092a
2018-10-23 05:46:23 INFO ClientCnxn:975 - Opening socket connection to server ZK1/ZK1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-23 05:46:23 INFO ClientCnxn:852 - Socket connection established to ZK1/ZK1:2181, initiating session
2018-10-23 05:46:23 INFO ClientCnxn:1235 - Session establishment complete on server ZK1/ZK1:2181, sessionid = 0x1667e7f79ba0087, negotiated timeout = 60000
2018-10-23 05:46:23 INFO ConnectionStateManager:228 - State change: CONNECTED
2018-10-23 05:46:23 INFO ZooKeeperHiveClientHelper:83 - Selected HiveServer2 instance with uri: hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server2;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM
2018-10-23 05:46:23 INFO ZooKeeper:684 - Session: 0x1667e7f79ba0087 closed
2018-10-23 05:46:23 INFO ClientCnxn:512 - EventThread shut down
2018-10-23 05:46:23 INFO HiveConnection:227 - Will retry opening client transport
2018-10-23 05:46:23 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO HiveConnection:208 - Could not open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO CuratorFrameworkImpl:224 - Starting
2018-10-23 05:46:23 INFO ZooKeeper:438 - Initiating client connection, connectString=ZK1:2181,ZK2:2181,ZK3:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2133814f
2018-10-23 05:46:23 INFO ClientCnxn:975 - Opening socket connection to server ZK2/ZK2:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-23 05:46:23 INFO ClientCnxn:852 - Socket connection established to ZK2/ZK2:2181, initiating session
2018-10-23 05:46:23 INFO ClientCnxn:1235 - Session establishment complete on server ZK2/ZK2:2181, sessionid = 0x2667e7f0d77007c, negotiated timeout = 60000
2018-10-23 05:46:23 INFO ConnectionStateManager:228 - State change: CONNECTED
2018-10-23 05:46:23 INFO ZooKeeper:684 - Session: 0x2667e7f0d77007c closed
2018-10-23 05:46:23 INFO ClientCnxn:512 - EventThread shut down
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper (state=08S01,code=0)
0: jdbc:hive2://ZK1:2181,Hive-Server2 (closed)> [hive@Remore-Server ~]$

ZKCLI output
[zk Hive-server1:2181(CONNECTED) 0] ls /
[hive, cluster, brokers, infra-solr, hbase-unsecure, kafka-acl, kafka-acl-changes, admin, isr_change_notification, log_dir_event_notification, accumulo, rmstore, hbase-secure, consumers, latest_producer_id_block, registry, controller, storm, zookeeper, yarn-leader-election, tracers, hadoop-ha, controller_epoch, hiveserver2, druid, ambari-metrics-cluster, config]
[zk: Hive-server1:2181(CONNECTED) 1]

2) I also enabled SSL in Hive, checked the logs of hiveserevr2 and it states "
INFO [Thread-19]: auth.HiveAuthFactory (HiveAuthFactory.java:getServerSSLSocket(293)) - SSL Server Socket Enabled Protocols: [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]
"

[hive@Hive-Server1 ~]$ beeline
Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive
beeline> !connect jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;
Connecting to jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;
Enter username for jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;:
Enter password for jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;:
18/10/23 06:22:00 [main]: WARN jdbc.HiveConnection: Failed to connect to Hive-Server1:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
0: jdbc:hive2://Hive-Server1:10000/default (closed)> [hive@Hive-Server1 ~]$
[hive@Hive-Server1 ~]$

Followed below steps to generate keys & import/export certs
keytool -genkey -alias Hive-Server1 -keyalg RSA -keystore hive-keystore.jks -keysize 2048
keytool -list -keystore hive-keystore.jks
keytool -export -alias Hive-Server1 -file hive.crt -keystore hive-keystore.jks
keytool -import -trustcacerts -alias Hive-Server1 -file hive.crt -keystore hive-truststore.jks
keytool -list -keystore hive-truststore.jks Configs - SSL is enabled + keystore file & password is also set and below was one of the error in hiveserver 2.log
2018-10-23 05:42:02,416 ERROR [HiveServer2-Handler-Pool: Thread-61]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
2018-10-23 05:42:02,456 ERROR [HiveServer2-Handler-Pool: Thread-62]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.

Appreciate your help

Thanks