Support Questions
Find answers, ask questions, and share your expertise

Hive beeline Connection string for Wire encryption SSL with Kerberos not working

Explorer

I am using HDP 2.6.5, HS2 is kerberized & HA enabled. remote ZK connection string is not working but locally Kerberos connection string works

Got stuck between Kerberos HS2 HA & SSL wire encryption for Hive. Need your help,
1) Before enabling SSL, Hive with Kerberos was working from local hiveserver2 beeline

[hive@server1 hive]$ beeline
Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive
beeline> !connect jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Connecting to jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Enter username for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
Enter password for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
Connected to: Apache Hive (version 1.2.1000.2.6.5.0-292)
Driver: Hive JDBC (version 1.2.1000.2.6.5.0-292)
Transaction isolation: TRANSACTION_REPEATABLE_READ

post SSL enabling:-

beeline> !connect jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Connecting to jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM
Enter username for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
Enter password for jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM:
18/10/23 06:43:05 [main]: WARN jdbc.HiveConnection: Failed to connect to server1-IP:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://server1-IP:10000/default;principal=hive/server1@EXAMPLE.COM: Invalid status 21 (state=08S01,code=0)
0: jdbc:hive2://server1-IP:10000/default (closed)>


Note - However the zookeeper connection string in HA mode establishes the connection and again it disconnects and was not working from remote server, see below error
I tried changing the transport mode from binary to http but that did not worked and reverted to binary.

beeline> !connect jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
Connecting to jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
Enter username for jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;:
Enter password for jdbc:hive2://ZK1:2181,ZK2:2181,ZK3:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;:
2018-10-23 05:46:22 INFO Utils:310 - Supplied authorities: ZK1:2181,ZK2:2181,ZK3:2181
2018-10-23 05:46:22 INFO CuratorFrameworkImpl:224 - Starting
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:host.name=Remore-Server
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.version=1.8.0_181
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.class.path=/usr/lib/python2.7/site-packages/pyspark/conf:/usr/lib/python2.7/site-packages/pyspark/jars/JavaEWAH-0.3.2.jar:/usr/lib/python2.7/site-packages/pyspark/jars/RoaringBitmap-0.5.11.jar:/usr/lib/python2.7/site-packages/pyspark/jars/ST4-4.0.4.jar:/usr/lib/python2.7/site-packages/pyspark/jars/activation-1.1.1.jar:/usr/lib/python2.7/site-packages/pyspark/jars/aircompressor-0.8.jar:/usr/lib/python2.7/site-packages/pyspark/jars/antlr-2.7.7.jar:/usr/lib/python2.7/site-packages/pyspark/jars/antlr-runtime-3.4.jar:/usr/lib/python2.7/site-packages/pyspark/jars/antlr4-runtime-4.7.jar:/usr/lib/python2.7/site-packages/pyspark/jars/aopalliance-1.0.jar:/usr/lib/python2.7/site-packages/pyspark/jars/aopalliance-repackaged-2.4.0-b34.jar:/usr/lib/python2.7/site-packages/pyspark/jars/apache-log4j-extras-1.2.17.jar:/usr/lib/python2.7/site-packages/pyspark/jars/apacheds-i18n-2.0.0-M15.jar:/usr/lib/python2.7/site-packages/pyspark/jars/apa............
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:java.compiler=<NA>
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:os.name=Linux
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:os.arch=amd64
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:os.version=3.10.0-693.el7.x86_64
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:user.name=hive
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:user.home=/home/hive
2018-10-23 05:46:22 INFO ZooKeeper:100 - Client environment:user.dir=/home/hive
2018-10-23 05:46:22 INFO ZooKeeper:438 - Initiating client connection, connectString=ZK1:2181,ZK2:2181,ZK3:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@3e77a1ed
2018-10-23 05:46:22 INFO ClientCnxn:975 - Opening socket connection to server ZK1/ZK1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-23 05:46:22 INFO ClientCnxn:852 - Socket connection established to ZK1/ZK1:2181, initiating session
2018-10-23 05:46:22 INFO ClientCnxn:1235 - Session establishment complete on server ZK1/ZK1:2181, sessionid = 0x1667e7f79ba0086, negotiated timeout = 60000
2018-10-23 05:46:22 INFO ConnectionStateManager:228 - State change: CONNECTED
2018-10-23 05:46:23 INFO ZooKeeperHiveClientHelper:83 - Selected HiveServer2 instance with uri: hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM
2018-10-23 05:46:23 INFO ZooKeeper:684 - Session: 0x1667e7f79ba0086 closed
2018-10-23 05:46:23 INFO ClientCnxn:512 - EventThread shut down
2018-10-23 05:46:23 INFO Utils:397 - Resolved authority: hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM
2018-10-23 05:46:23 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO HiveConnection:208 - Could not open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO CuratorFrameworkImpl:224 - Starting
2018-10-23 05:46:23 INFO ZooKeeper:438 - Initiating client connection, connectString=ZK1:2181,ZK2:2181,ZK3:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@7995092a
2018-10-23 05:46:23 INFO ClientCnxn:975 - Opening socket connection to server ZK1/ZK1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-23 05:46:23 INFO ClientCnxn:852 - Socket connection established to ZK1/ZK1:2181, initiating session
2018-10-23 05:46:23 INFO ClientCnxn:1235 - Session establishment complete on server ZK1/ZK1:2181, sessionid = 0x1667e7f79ba0087, negotiated timeout = 60000
2018-10-23 05:46:23 INFO ConnectionStateManager:228 - State change: CONNECTED
2018-10-23 05:46:23 INFO ZooKeeperHiveClientHelper:83 - Selected HiveServer2 instance with uri: hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server2;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM
2018-10-23 05:46:23 INFO ZooKeeper:684 - Session: 0x1667e7f79ba0087 closed
2018-10-23 05:46:23 INFO ClientCnxn:512 - EventThread shut down
2018-10-23 05:46:23 INFO HiveConnection:227 - Will retry opening client transport
2018-10-23 05:46:23 INFO HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO HiveConnection:208 - Could not open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth-conf;hive.server2.thrift.bind.host=Hive-Server1;hive.server2.thrift.port=10000;hive.server2.use.SSL=true;hive.server2.authentication.kerberos.principal=hive/_HOST@EXAMPLE.COM/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf;
2018-10-23 05:46:23 INFO CuratorFrameworkImpl:224 - Starting
2018-10-23 05:46:23 INFO ZooKeeper:438 - Initiating client connection, connectString=ZK1:2181,ZK2:2181,ZK3:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@2133814f
2018-10-23 05:46:23 INFO ClientCnxn:975 - Opening socket connection to server ZK2/ZK2:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-23 05:46:23 INFO ClientCnxn:852 - Socket connection established to ZK2/ZK2:2181, initiating session
2018-10-23 05:46:23 INFO ClientCnxn:1235 - Session establishment complete on server ZK2/ZK2:2181, sessionid = 0x2667e7f0d77007c, negotiated timeout = 60000
2018-10-23 05:46:23 INFO ConnectionStateManager:228 - State change: CONNECTED
2018-10-23 05:46:23 INFO ZooKeeper:684 - Session: 0x2667e7f0d77007c closed
2018-10-23 05:46:23 INFO ClientCnxn:512 - EventThread shut down
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper (state=08S01,code=0)
0: jdbc:hive2://ZK1:2181,Hive-Server2 (closed)> [hive@Remore-Server ~]$

ZKCLI output
[zk Hive-server1:2181(CONNECTED) 0] ls /
[hive, cluster, brokers, infra-solr, hbase-unsecure, kafka-acl, kafka-acl-changes, admin, isr_change_notification, log_dir_event_notification, accumulo, rmstore, hbase-secure, consumers, latest_producer_id_block, registry, controller, storm, zookeeper, yarn-leader-election, tracers, hadoop-ha, controller_epoch, hiveserver2, druid, ambari-metrics-cluster, config]
[zk: Hive-server1:2181(CONNECTED) 1]

2) I also enabled SSL in Hive, checked the logs of hiveserevr2 and it states "
INFO [Thread-19]: auth.HiveAuthFactory (HiveAuthFactory.java:getServerSSLSocket(293)) - SSL Server Socket Enabled Protocols: [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]
"

[hive@Hive-Server1 ~]$ beeline
Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive
beeline> !connect jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;
Connecting to jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;
Enter username for jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;:
Enter password for jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;:
18/10/23 06:22:00 [main]: WARN jdbc.HiveConnection: Failed to connect to Hive-Server1:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://Hive-Server1:10000/default;ssl=true;sslTrustStore=/etc/hive/conf/certs/hive-truststore.jks;trustStorePassword=test1234;: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
0: jdbc:hive2://Hive-Server1:10000/default (closed)> [hive@Hive-Server1 ~]$
[hive@Hive-Server1 ~]$

Followed below steps to generate keys & import/export certs
keytool -genkey -alias Hive-Server1 -keyalg RSA -keystore hive-keystore.jks -keysize 2048
keytool -list -keystore hive-keystore.jks
keytool -export -alias Hive-Server1 -file hive.crt -keystore hive-keystore.jks
keytool -import -trustcacerts -alias Hive-Server1 -file hive.crt -keystore hive-truststore.jks
keytool -list -keystore hive-truststore.jks Configs - SSL is enabled + keystore file & password is also set and below was one of the error in hiveserver 2.log
2018-10-23 05:42:02,416 ERROR [HiveServer2-Handler-Pool: Thread-61]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1849)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
at sun.security.ssl.InputRecord.read(InputRecord.java:527)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more
2018-10-23 05:42:02,456 ERROR [HiveServer2-Handler-Pool: Thread-62]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.

Appreciate your help

Thanks

0 REPLIES 0