question Re: Hiveserver2 HA using haproxy load balancing in Support Questions

Hiveserver2 HA using haproxy load balancing

VijayM — Fri, 16 Sep 2022 13:47:06 GMT

Hello Team,

We have CDH 5.15 cluster running and have kerberos and TLS enabled for all services in the cluster.

We would like to enable for Hiveserver2 using haproxy load balancer.

We have enable HA for hivemetastore using below link. 2 instance of hive metastore is up and running.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hivemetastore.html

Refering below link for hiveserver2 ha.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hiveserver2.html

haproxy, 1 instance of hive metastore, 1 instance of hiveserver2 installed on same node.

beeline throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxxx;principal=hive/aabc@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

Below snap for haproxy config

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
listen hiveserver2 :10001
mode tcp
option tcplog
balance source
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

Kindly suggest?

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Mon, 08 Oct 2018 17:03:26 GMT

@VijayM,

We see by the following error that the failure occurred during the TLS handshake:

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

In this case, it is probably that the server did not understand the connection sent to it. In order to debug further, you can examine the logs of your HAProxy and also the HiveServer2 instance that you connected to.

I would also suggest testing without the HAProxy (connect directly with beeline to each of the HS2 instances and see if you can connect. This will help isolate whether to look more closely at HiveServer2 or the HAProxy.

If you know tcpdump, it is perfect for debugging TLS handshake problems since it lets you see all the handshake communication. Wireshark can decode the packets and display the handshake nicely. If that is not something you know well, let's hit the logs first.

Re: Hiveserver2 HA using haproxy load balancing

Rebecca784 — Thu, 11 Oct 2018 07:38:48 GMT

@VijayM wrote:
Hello Team,

We have CDH 5.15 cluster running and have kerberos and TLS enabled for all services in the cluster.

We would like to enable for Hiveserver2 using haproxy load balancer.

We have enable HA for hivemetastore using below link. 2 instance of hive metastore is up and running.
https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hivemetastore.html

Refering below link for hiveserver2 ha.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hiveserver2.html

haproxy, 1 instance of hive metastore, 1 instance of hiveserver2 installed krogerfeedback on same node.

beeline throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxxx;principal=hive/aabc@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

Below snap for haproxy config

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
listen hiveserver2 :10001
mode tcp
option tcplog
balance source
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

Kindly suggest?

- Vijay M

This is getting really complicated for me, please help!

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Tue, 16 Oct 2018 15:35:10 GMT

@bgooley,

I have TLS enabled hiveserver2 with 2 instance running on 2 different hosts.

haproxy installed and configured on same server where 1 hive instance running.

Kindly confirm below.

1. DO i need to define TLS cert anywhere in haproxy config, If yes any documentation for it?

2. Does haproxy also needs to be configured with TLS?

Any documentation for installing and conifuring load balancer for TLS enabled hiveserver2.

- VIjay Mishra

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Wed, 17 Oct 2018 18:04:14 GMT

@VijayM,

If you are using TLS passthrough, then you don't need to configure certificates fo HAProxy as the TLS handshake is done with the HS2 servers themselves. This does add some extra work for you, though, as it means that you need to be sure that the hostname(s) in the HS2 server certificates match the name of your HAProxy host.

This can be done in a few ways, such as issuing a server certificate that contains SubjectAltName value equal to the HAProxy host's fully-qualified domain name or you could use a wildcard that matches the domain.

If you are using TLS termination where the client will do the TLS handshake with HAProxy and then can either do TLS or non-TLS connections to backend servers. In this case, HAProxy will decrypt the incoming request and then re-encrypt it if your HS2 servers are listening on TLS ports.

In that case, you do have to specify a server certificate for HAProxy's frontend and you need to use a trust store to trust the signer of the HS2 certificates.

There is information out there, but this page (dispite a few mistakes) is pretty good talking about each:

https://serversforhackers.com/c/using-ssl-certificates-with-haproxy

An example of pass-through is one I'm using on my server:

frontend hiveserver2_front
bind *:10015 ssl crt /etc/cdep-ssl-conf/CA_STANDARD/cert_key.pem
mode tcp
option tcplog
default_backend hiveserver2

backend hiveserver2
balance source
mode tcp
server hiveserver2_1 tls12-1.example.com:10000 ssl ca-file /etc/cdep-ssl-conf/CA_STANDARD/truststore.pem
server hiveserver2_2 tls12-4.example.com:10000 ssl ca-file /etc/cdep-ssl-conf/CA_STANDARD/truststore.pem
server hiveserver2_3 tls12-2.example.com:10000 ssl ca-file /etc/cdep-ssl-conf/CA_STANDARD/truststore.pem

NOTE: in the above, I have mode tcp set which means I'm using passthrough (no http header evaluation and therefore no need to decrypt)

Since I have server and truststore files configured, though, I could switch to mode http and do termination at the HAProxy.

I'm no HAProxy expert, but I am pretty sure the above should help you.

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Thu, 01 Nov 2018 11:35:04 GMT

@bgooley,

I was engaged in some other projects so unable to reply on it. Started working on it today.

When iam connecting both hiveserver2 instance without haproxy, removed the load balance entry from hive confiugration.

I am able to connect to both hiveserver2 instance from beeline.

I have hiveserver2 TLS enable using CA signed certificates. and Hiveserver2 certificates are in Java format i.e. .jks(keystore.jks and truststore.jks).

In my haproxy configuration at bind line i am giving keystore.jks entry and for backend entry i am giving truststore.jks entry for both server.

Kindly confirm is it correct or suggest?

- VIjay M

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Tue, 06 Nov 2018 23:27:54 GMT

Hi @VijayM,

Without seeing the configuration you have, it is hard to say what is correct. Perhaps you can share and we can see if there is something obvois. I would strongly suggest looking at the HAProxy logs an the HiveServer2 logs when the problem happens to look for any TLS errors or related messages.

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Wed, 07 Nov 2018 04:12:52 GMT

@bgooley,

Haproxy log doesn't shows anything and even Hiveserver2 logs.

Will send you configuration post Monday as I am on leave.

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Tue, 13 Nov 2018 08:45:32 GMT

@bgooley

Below find details of certificates which i have on cluster.

below certificate is from root CA

-rwxr-xr-x. 1 cloudera-scm cloudera-scm 8152 Oct 5 10:36 cacerts.pem

Below certificate are keystore and trustore used by Hive service TLS enabled.

-rwxr-xr-x. 1 cloudera-scm cloudera-scm 9624 Oct 5 10:38 cloudera_keystore.jks
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 4048 Oct 5 10:39 cloudera_truststore.jks

Below find configuration of haproxy

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend hiveserver2_front
bind *:443
option tcplog
mode tcp
default_backend hiveserver2

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
backend hiveserver2
mode tcp
balance source
option ssl-hello-chk
server hiveserver2_1 abc:10000 check
server hiveserver2_2 xyz:10000 check

--- hive server2 configuration from cloudera manager configured with below property

HiveServer2 Load Balancer abc:443

1. Kindly confirm does in above property do i have to add https or http? Is it require?

2. Kindly review the configuration and let me know if anything more details require?

3. one of hiveserver2 instance and haproxy services configured on same server i.e. abc, Is it an issue?

Kindly suggest?

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Wed, 14 Nov 2018 20:37:46 GMT

@VijayM,

Based on your original message and your configuration, I think the HAProxy bind port is the issue.

You have:

bind *:443

But you are trying to connect via TLS to port 10001

Maybe try:

bind *:10001

Then restart HAProxy.

Hope it is that simple. If that doesn't work, let us know and we can use openssl s_client to observe the handshake to see what happens.

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Thu, 15 Nov 2018 05:54:28 GMT

@bgooley

Same issue with port 10001 or port 443.

Below snap confirms haproxy started and running on port 10001.

[root@abc ~]# ps -ef | grep -i haproxy
root 2620129 1 0 06:19 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 2620130 2620129 0 06:19 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy 2620131 2620130 0 06:19 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

[root@abc ~]# netstat -tunlp | grep 10001
tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN 2620131/haproxy
[root@abc ~]#

Below are 2 scenarios which i tried and explained. Kindly check and suggest to fix.

Case1:

When i removed haproxy load balancer porperty from hive configuration and trying to connect individual haproxy services through beeline. i am able to connect. Below snap for the same.

beeline> !connect jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:1>

beeline> !connect jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://a301-8883-2675.gdzd.ubs.net:1>

Case2:

With haproxy load balancer property in hive configuration with port 10001 configured in haproxy configuration its not working and throws error.

No logs of both hiveserver2 instance and haproxy gets updated for above error.

with above scnario when i am trying to connect individual hiveserver2 instances i am able to connect to hiveserver2 instance on haproxy running but unable to connect to other hiveserver2 instance and gets TLS error.

Below snap for both.

-- Successfully able to connect to hiveserrver2 where haproxy also running.

--- Unable to connect to other hiveserver2 instance

beeline> !connect jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: Peer indicated failure: GSS initiate failed (state=08S01,code=0)
beeline>

hiveserver2 log shows below error.

2018-11-15 06:46:08,217 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-40]: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)

Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167)
... 14 more
Caused by: KrbException: Checksum failed

2018-11-15 06:46:08,220 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-40]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: GSS initiate failed

Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Thu, 15 Nov 2018 07:15:35 GMT

@bgooley

Also if i configure haproxy using below configuration its does not starts.

frontend hiveserver2_front
bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem
option tcplog
mode tcp
default_backend hiveserver2

below ask for passhphrase, not sure why

[root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Enter PEM pass phrase:

When i entered the password/passphrase it gives invalid configuration file

[root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Enter PEM pass phrase:
[WARNING] 318/081440 (2708462) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Configuration file is valid

Kindly suggest what wrong i m doing ?

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Thu, 15 Nov 2018 16:44:20 GMT

@VijayM,

The configuration you are using is not correct as it is a mix of pass-through and termination.

You can remove everything from ssl onward in line:

bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem

so it becomes:

bind *:10001

I looked back at my first post and it appears I made a mistake when pasting and forgot to remove the "ssl" part from my pass-through example. Sorry for the confusion.

NOTE:

If you are doing TLS termination, then being prompted for the key password is expected if you have a key file that is password protected.

NOTE2:

In order to get rid of that WARNING if you want to use termination, add tune.ssl.default-dh-param 2048 to the "global" section of your haproxy.cfg and restart.

In ordre to debug the javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0) issue we really need to see why the handshake is being terminated. If the log of haproxy and both hiveserver2 servers don't show any TLS messages at the time of the failure, then the next best thing is to do a packet capture on port host where the beeline is run and also on the HiveServer2 server.

Since the TLS handshake is done in the clear, a packet capture can be opened in WireShark where the handshake will be evident.

For example:

1 - runon the beeline host:

# tcpdump -i any -w ~/beeline.pcap port 10001

2 - run on the HiveServer2 host (shut down one so that the load balancer must choose one and you know which):

# tcpdump -i any -w ~/hs2.pcap port 10000

3 - run the beeline command so that it fails

4 - Ctrl-c both tcpdumps

5 - open the pcap files in Wireshark. You may need to use "decode as..." to decode the 10001 and 10000 ports as SSL/TLS in order to see the TLS handshake.

If you are unfamiliar with packet capture/wireshark, then try this:

# openssl s_client -connect <load_balancer_host>:10001 -msg -debug

This will have openssl client print out the handshake process via the load balancer.

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Thu, 15 Nov 2018 18:11:52 GMT

Hello bgooley,

Sure, let me try what u suggested.

I would like to use Ssl pass through in haproxy config.

Kindly provide exact configuration needs to define in haproxy config.

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Fri, 16 Nov 2018 12:14:55 GMT

@bgooley,

Kindly find below updated hzproxy configuration.

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
backend hiveserver2
mode tcp
balance source
#option ssl-hello-chk
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

Updated hiveserver2 configuration with hive load balancer.

HiveServer2 Load Balancer - abc:10001

Tried to connect through beeline but it still gives the same TLS error.

Tried to test TLS connectivity with openssl command which you suggested. below snap for the same. Kindly check and suggest.

[root@abc ~]# openssl s_client -connect abc:10001 -msg -debug
CONNECTED(00000003)
write to 0x14e0e00 [0x14f73b0] (289 bytes => 289 (0x121))
0000 - 16 03 01 01 1c 01 00 01-18 03 03 f2 83 c8 e8 a0 ................
0010 - 24 eb da e4 31 75 05 76-62 91 34 b1 4b cc ce 21 $...1u.vb.4.K..!
0020 - 15 99 d7 cc 76 93 78 7d-5e bb 8f 00 00 ac c0 30 ....v.x}^......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........
00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................
00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C..
00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................
00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0120 - 01 .
>>> TLS 1.2 [length 0005]
16 03 01 01 1c
>>> TLS 1.2 Handshake [length 011c], ClientHello
01 00 01 18 03 03 f2 83 c8 e8 a0 24 eb da e4 31
75 05 76 62 91 34 b1 4b cc ce 21 15 99 d7 cc 76
93 78 7d 5e bb 8f 00 00 ac c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00
6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00
87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00
40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00
99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0
2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00
96 00 41 c0 12 c0 08 00 16 00 13 00 10 00 0d c0
0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00
05 00 04 00 ff 01 00 00 43 00 0b 00 04 03 00 01
02 00 0a 00 0a 00 08 00 17 00 19 00 18 00 16 00
23 00 00 00 0d 00 20 00 1e 06 01 06 02 06 03 05
01 05 02 05 03 04 01 04 02 04 03 03 01 03 02 03
03 02 01 02 02 02 03 00 0f 00 01 01
read from 0x14e0e00 [0x14fc910] (7 bytes => 0 (0x0))
140683941742480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1542369743
Timeout : 300 (sec)
Verify return code: 0 (ok)

Tried to shutdown 1 of hiveserver2 instance and tested with beeline but still same issue.

Hiveserver2 TLS certificates are in JKS format.

Kindly suggest.

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Fri, 16 Nov 2018 17:27:39 GMT

@VijayM,

The openssl debug information indicates that the client makes a connection to a server but the server does not return a certificate. Since a direct connection to HiveServer2 does not have the problem, I conclude that your haproxy is still using termination even though your configuration snippet would indicate otherwise. Based on what you have provided it appears:

1. your connection to port 10001 is using TLS termination at the haproxy

2. the server certificate is not valid so no TLS handshake can be performed.

Basically, the configuration you show cannot be the one that is being used for haproxy that is running and listening on port 10001 so perhaps it was not restarted.

openssl s_client will return the following error if the port it connects to is not listening on TLS:

139972358285128:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:769:

Since you are seeing:

SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177

That indicates there was an actual problem on the server side. The server in this case must be your haproxy.

So, I think it would be good to list the full haproxy configuration file and also make sure that it really did restart since your last change. I used your config file and pass-through TLS worked perfectly to my HS2 servers.

I think we must be fighting an haproxy config/restart issue since the frontend/backend you showed last worked for me. I actually copied and pasted your config and changed the hostnames only.

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Fri, 16 Nov 2018 17:42:52 GMT

Hello bgooley,

I will going to office on Tuesday now, will do the troubleshooting on that
day and if still the same issue can provide you full haproxy config file.

If u can provide ur full haproxy config file then I will compare and will
correct the things.

- Vijay M

Re: Hiveserver2 HA using haproxy load balancing

bgooley — Fri, 16 Nov 2018 18:03:14 GMT

global

log 127.0.0.1 local2

pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /tmp/haproxy
tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout check 10s
maxconn 3000

listen admin
bind *:8000
stats enable

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance source
server hs2_1 host1.example.com:10000
server hs2_2 host2.example.com:10000

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Mon, 19 Nov 2018 06:52:39 GMT

@bgooley

I have updated haproxy configuration as you provided. Below snap for the same.

global

log 127.0.0.1 local2

pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# stats socket /tmp/haproxy
# tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
# option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout check 10s
maxconn 3000

listen admin
bind *:8000
stats enable

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance source
server hs2_1 abc:10000
server hs2_2 xyz:10000

restarted haproxy and it started with current timestamp. Check and verified.

Below find configuration info

haproxy service and 1st hiveserver instance running on server : abc
2nd hiveserver instance running on server : xyz

Scenario 1:

Both hiveserver2 instance up and running.
Connecting from beeline from server abc, throws below error.

Connecting from beeline from server xyz is successful, Below snap for the same.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://abc:1> show databases;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://abc:1> show databases;
INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds
INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds
INFO : OK
+----------------+--+
| database_name |
+----------------+--+
| default |
| test1 |
| test |
+----------------+--+

Scenario 2:

hiveserver2 instance stopped on server xyz

Connecting from beeline from server abc, throws below error.

Connecting from beeline from server xyz is successful, Below snap for the same.

Scenario 3:

hiveserver2 instance stopped on server abc

Connecting from beeline from server abc, throws below error.

Connecting from beeline from server xyz is failed, Below snap for the same. Request from haproxy seems not getting forwarded to xyzs server when abc is not available.

beeline> !connect jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
scan complete in 1ms
Connecting to jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

Kindly check for all above scenario and suggest.

- VIjay M

Re: Hiveserver2 HA using haproxy load balancing

VijayM — Mon, 19 Nov 2018 07:51:07 GMT

@bgooley

Did some more troubleshooting.

Updated alogrithm for load balance to roundrobin from source as mentioned below in haproxy configuration and started to get errors in hiveserver2 logs.

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance roundrobin
server hs2_1 a301-8883-0447.gdzd.ubs.net:10000 check
server hs2_2 a301-8883-2675.gdzd.ubs.net:10000 check

When tried to connect from beeline from any server my 2nd connection always to server xyz and connection getting failed and hiveserver2 logs of xyz server throws below error.

2018-11-19 08:48:05,964 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-44]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:794)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:791)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1904)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:791)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more

Kindly suggest ?

- Vijay M