Reply
Explorer
Posts: 10
Registered: ‎07-05-2018

Hiveserver2 HA using haproxy load balancing

[ Edited ]

Hello Team,

 

We have CDH 5.15 cluster running and have kerberos and TLS enabled for all services in the cluster.

 

We would like to enable for Hiveserver2 using haproxy load balancer.

 

We have enable HA for hivemetastore using below link. 2 instance of hive metastore is up and running.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hivemetastore.html

 

Refering below link for hiveserver2 ha.

 

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hiveserver2.html

 

haproxy, 1 instance of hive metastore, 1 instance of hiveserver2 installed on same node.

 

beeline throws below error.

 

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxxx;principal=hive/aabc@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

 

 

Below snap for haproxy config

 

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
listen hiveserver2 :10001
mode tcp
option tcplog
balance source
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

 

 

Kindly suggest?

 

 

- Vijay M

Posts: 910
Topics: 1
Kudos: 208
Solutions: 111
Registered: ‎04-22-2014

Re: Hiveserver2 HA using haproxy load balancing

@VijayM,

 

We see by the following error that the failure occurred during the TLS handshake:

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

 

In this case, it is probably that the server did not understand the connection sent to it.  In order to debug further, you can examine the logs of your HAProxy and also the HiveServer2 instance that you connected to.

 

I would also suggest testing without the HAProxy (connect directly with beeline to each of the HS2 instances and see if you can connect.  This will help isolate whether to look more closely at HiveServer2 or the HAProxy.

 

If you know tcpdump, it is perfect for debugging TLS handshake problems since it lets you see all the handshake communication.  Wireshark can decode the packets and display the handshake nicely. If that is not something you know well, let's hit the logs first.

Highlighted
New Contributor
Posts: 2
Registered: ‎09-07-2018

Re: Hiveserver2 HA using haproxy load balancing

[ Edited ]

@VijayM wrote:

Hello Team,

 

We have CDH 5.15 cluster running and have kerberos and TLS enabled for all services in the cluster.

 

We would like to enable for Hiveserver2 using haproxy load balancer.

 

We have enable HA for hivemetastore using below link. 2 instance of hive metastore is up and running.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hivemetastore.html

 

Refering below link for hiveserver2 ha.

 

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hiveserver2.html

 

haproxy, 1 instance of hive metastore, 1 instance of hiveserver2 installed krogerfeedback on same node.

 

beeline throws below error.

 

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxxx;principal=hive/aabc@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

 

 

Below snap for haproxy config

 

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
listen hiveserver2 :10001
mode tcp
option tcplog
balance source
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

 

 

Kindly suggest?

 

 

- Vijay M


This is getting really complicated for me, please help!

Announcements