Reply
Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

@bgooley

 

Same issue with port 10001 or port 443.

 

Below snap confirms haproxy started and running on port 10001.

 

[root@abc ~]# ps -ef | grep -i haproxy
root 2620129 1 0 06:19 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 2620130 2620129 0 06:19 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy 2620131 2620130 0 06:19 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

[root@abc ~]# netstat -tunlp | grep 10001
tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN 2620131/haproxy
[root@abc ~]#

 

 

Below are 2 scenarios which i tried and explained. Kindly check and suggest to fix.

 

 

Case1:  

 

When i removed haproxy load balancer porperty from hive configuration and trying to connect individual haproxy services through beeline. i am able to connect. Below snap for the same.

 

beeline> !connect jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:1>

 

 

 

beeline> !connect jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://a301-8883-2675.gdzd.ubs.net:1>

 

 

 

 

Case2:

 

With haproxy load balancer property in hive configuration with port 10001 configured in haproxy configuration its not working and throws error.

 

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)
beeline>

 

 

 

No logs of both hiveserver2 instance and haproxy gets updated for above error.

 

 

with above scnario when i am trying to connect individual hiveserver2 instances i am able to connect to hiveserver2 instance on haproxy running but unable to connect to other hiveserver2 instance and gets TLS error. 

 

Below snap for both.

 

--  Successfully able to connect to hiveserrver2 where haproxy also running.

 

beeline> !connect jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:1>

 

 

--- Unable to connect to other hiveserver2 instance

 

beeline> !connect jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: Peer indicated failure: GSS initiate failed (state=08S01,code=0)
beeline>

 

 

hiveserver2 log shows below error.

 

2018-11-15 06:46:08,217 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-40]: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)

 

Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167)
... 14 more
Caused by: KrbException: Checksum failed

 

2018-11-15 06:46:08,220 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-40]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: GSS initiate failed

 

Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed

 

 

- Vijay M

Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

@bgooley

 

Also if i configure haproxy using below configuration its does not starts.

 

frontend hiveserver2_front
bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem
option tcplog
mode tcp
default_backend hiveserver2

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
backend hiveserver2
mode tcp
balance source
option ssl-hello-chk
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

 

below ask for passhphrase, not sure why

 

[root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Enter PEM pass phrase:

 

When i entered the password/passphrase it gives invalid configuration file

 

[root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Enter PEM pass phrase:
[WARNING] 318/081440 (2708462) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Configuration file is valid

 

 

Kindly suggest what wrong i m doing ?

 

- Vijay M

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: Hiveserver2 HA using haproxy load balancing

@VijayM,

 

The configuration you are using is not correct as it is a mix of pass-through and termination.

 

You can remove everything from ssl onward in line:

bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem

so it becomes:

bind *:10001

 

I looked back at my first post and it appears I made a mistake when pasting and forgot to remove the "ssl" part from my pass-through example.  Sorry for the confusion.

 

NOTE:

If you are doing TLS termination, then being prompted for the key password is expected if you have a key file that is password protected.

NOTE2:

In order to get rid of that WARNING if you want to use termination, add tune.ssl.default-dh-param 2048 to the "global" section of your haproxy.cfg and restart.

 

In ordre to debug the javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0) issue we really need to see why the handshake is being terminated.  If the log of haproxy and both hiveserver2 servers don't show any TLS messages at the time of the failure, then the next best thing is to do a packet capture on port host where the beeline is run and also on the HiveServer2 server.

 

Since the TLS handshake is done in the clear, a packet capture can be opened in WireShark where the handshake will be evident.

 

For example:

 

1 - runon the beeline host:

# tcpdump -i any -w ~/beeline.pcap port 10001

 

2 - run on the HiveServer2 host (shut down one so that the load balancer must choose one and you know which):

# tcpdump -i any -w ~/hs2.pcap port 10000

 

3 - run the beeline command so that it fails

 

4 - Ctrl-c both tcpdumps

 

5 - open the pcap files in Wireshark.  You may need to use "decode as..." to decode the 10001 and 10000 ports as SSL/TLS in order to see the TLS handshake.

 

If you are unfamiliar with packet capture/wireshark, then try this:

 

# openssl s_client -connect <load_balancer_host>:10001 -msg -debug

 

This will have openssl client print out the handshake process via the load balancer.

 

Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

Hello bgooley,

Sure, let me try what u suggested.

I would like to use Ssl pass through in haproxy config.

Kindly provide exact configuration needs to define in haproxy config.

- Vijay M
Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

@bgooley,

 

Kindly find below updated hzproxy configuration.

 

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
backend hiveserver2
mode tcp
balance source
#option ssl-hello-chk
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

 

Updated hiveserver2 configuration with hive load balancer.

 

HiveServer2 Load Balancer  -  abc:10001

 

 

Tried to connect through beeline but it still gives the same TLS error.

 

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

 

 

Tried to test TLS connectivity with openssl command which you suggested. below snap  for the same. Kindly check and suggest.

 

[root@abc ~]# openssl s_client -connect abc:10001 -msg -debug
CONNECTED(00000003)
write to 0x14e0e00 [0x14f73b0] (289 bytes => 289 (0x121))
0000 - 16 03 01 01 1c 01 00 01-18 03 03 f2 83 c8 e8 a0 ................
0010 - 24 eb da e4 31 75 05 76-62 91 34 b1 4b cc ce 21 $...1u.vb.4.K..!
0020 - 15 99 d7 cc 76 93 78 7d-5e bb 8f 00 00 ac c0 30 ....v.x}^......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........
00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................
00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C..
00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................
00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0120 - 01 .
>>> TLS 1.2 [length 0005]
16 03 01 01 1c
>>> TLS 1.2 Handshake [length 011c], ClientHello
01 00 01 18 03 03 f2 83 c8 e8 a0 24 eb da e4 31
75 05 76 62 91 34 b1 4b cc ce 21 15 99 d7 cc 76
93 78 7d 5e bb 8f 00 00 ac c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00
6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00
87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00
40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00
99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0
2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00
96 00 41 c0 12 c0 08 00 16 00 13 00 10 00 0d c0
0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00
05 00 04 00 ff 01 00 00 43 00 0b 00 04 03 00 01
02 00 0a 00 0a 00 08 00 17 00 19 00 18 00 16 00
23 00 00 00 0d 00 20 00 1e 06 01 06 02 06 03 05
01 05 02 05 03 04 01 04 02 04 03 03 01 03 02 03
03 02 01 02 02 02 03 00 0f 00 01 01
read from 0x14e0e00 [0x14fc910] (7 bytes => 0 (0x0))
140683941742480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1542369743
Timeout : 300 (sec)
Verify return code: 0 (ok)

 

 

Tried to shutdown 1 of hiveserver2 instance and tested with beeline but still same issue.

 

Hiveserver2 TLS certificates are in JKS format.

 

Kindly suggest.

 

- Vijay M

 

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: Hiveserver2 HA using haproxy load balancing

@VijayM,

 

The openssl debug information indicates that the client makes a connection to a server but the server does not return a certificate.  Since a direct connection to HiveServer2 does not have the problem, I conclude that your haproxy is still using termination even though your configuration snippet would indicate otherwise.  Based on what you have provided it appears:

 

1. your connection to port 10001 is using TLS termination at the haproxy

2. the server certificate is not valid so no TLS handshake can be performed.

 

Basically, the configuration you show cannot be the one that is being used for haproxy that is running and listening on port 10001 so perhaps it was not restarted.

 

openssl s_client will return the following error if the port it connects to is not listening on TLS:

 

139972358285128:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:769:

 

Since you are seeing:

 

SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177

 

That indicates there was an actual problem on the server side.  The server in this case must be your haproxy.

 

So, I think it would be good to list the full haproxy configuration file and also make sure that it really did restart since your last change.  I used your config file and pass-through TLS worked perfectly to my HS2 servers.

 

I think we must be fighting an haproxy config/restart issue since the frontend/backend you showed last worked for me. I actually copied and pasted your config and changed the hostnames only.

Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

Hello bgooley,

I will going to office on Tuesday now, will do the troubleshooting on that
day and if still the same issue can provide you full haproxy config file.

If u can provide ur full haproxy config file then I will compare and will
correct the things.

- Vijay M
Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: Hiveserver2 HA using haproxy load balancing

global


   log 127.0.0.1 local2

   pidfile /var/run/haproxy.pid
   maxconn 4000
   user haproxy
   group haproxy
   daemon
   stats socket /tmp/haproxy
   tune.ssl.default-dh-param 2048

defaults
   mode http
   log global
   option httplog
   option dontlognull
   option forwardfor except 127.0.0.0/8
   option redispatch
   retries 3
   timeout http-request 10s
   timeout queue 1m
   timeout connect 10s
   timeout client 10m
   timeout server 10m
   timeout check 10s
   maxconn 3000

 

listen admin
   bind *:8000
   stats enable

 

frontend hiveserver2_front
   bind *:10001
   option tcplog
   mode tcp
   default_backend hiveserver2

backend hiveserver2
   mode tcp
   balance source
   server hs2_1 host1.example.com:10000
   server hs2_2 host2.example.com:10000

Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

 

@bgooley

 

I have updated haproxy configuration as you provided. Below snap for the same.

global


log 127.0.0.1 local2

pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# stats socket /tmp/haproxy
# tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
# option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout check 10s
maxconn 3000

 

listen admin
bind *:8000
stats enable

 

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance source
server hs2_1 abc:10000
server hs2_2 xyz:10000


restarted haproxy and it started with current timestamp. Check and verified.


Below find configuration info

haproxy service and 1st hiveserver instance running on server : abc
2nd hiveserver instance running on server : xyz


Scenario 1:

Both hiveserver2 instance up and running.
Connecting from beeline from server abc, throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)


Connecting from beeline from server xyz is successful, Below snap for the same.


beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://abc:1> show databases;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://abc:1> show databases;
INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds
INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds
INFO : OK
+----------------+--+
| database_name |
+----------------+--+
| default |
| test1 |
| test |
+----------------+--+

 

Scenario 2:

hiveserver2 instance stopped on server xyz


Connecting from beeline from server abc, throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)

 

Connecting from beeline from server xyz is successful, Below snap for the same.


beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://abc:1> show databases;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://abc:1> show databases;
INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds
INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds
INFO : OK
+----------------+--+
| database_name |
+----------------+--+
| default |
| test1 |
| test |
+----------------+--+

 


Scenario 3:

hiveserver2 instance stopped on server abc


Connecting from beeline from server abc, throws below error.


beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)


Connecting from beeline from server xyz is failed, Below snap for the same. Request from haproxy seems not getting forwarded to xyzs server when abc is not available.


beeline> !connect jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
scan complete in 1ms
Connecting to jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

 

 

Kindly check for all above scenario and suggest.


- VIjay M

Explorer
Posts: 29
Registered: ‎07-05-2018

Re: Hiveserver2 HA using haproxy load balancing

@bgooley

 

Did some more troubleshooting. 

 

Updated alogrithm for load balance to roundrobin from source as mentioned below in haproxy configuration and started to get errors in hiveserver2 logs.

 

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance roundrobin
server hs2_1 a301-8883-0447.gdzd.ubs.net:10000 check
server hs2_2 a301-8883-2675.gdzd.ubs.net:10000 check

 

When tried to connect from beeline from any server my 2nd connection always to server xyz and connection getting failed and hiveserver2 logs of xyz server throws below error.

 

2018-11-19 08:48:05,964 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-44]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:794)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:791)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1904)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:791)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more

 

 

 

Kindly suggest ?

 

 

- Vijay M

Announcements