Hiveserver2 HA using haproxy load balancing

[#1]

Hello Team,

We have CDH 5.15 cluster running and have kerberos and TLS enabled for all services in the cluster.

We would like to enable for Hiveserver2 using haproxy load balancer.

We have enable HA for hivemetastore using below link. 2 instance of hive metastore is up and running.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hivemetastore.html

Refering below link for hiveserver2 ha.

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_ha_hiveserver2.html

haproxy, 1 instance of hive metastore, 1 instance of hiveserver2 installed on same node.

beeline throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxx;principal=hive/aabc@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=xxxxxx;principal=hive/aabc@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

Below snap for haproxy config

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
listen hiveserver2 :10001
mode tcp
option tcplog
balance source
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

Kindly suggest?

- Vijay M

[#2]

@bgooley

Also if i configure haproxy using below configuration its does not starts.

frontend hiveserver2_front
bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem
option tcplog
mode tcp
default_backend hiveserver2

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
backend hiveserver2
mode tcp
balance source
option ssl-hello-chk
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

below ask for passhphrase, not sure why

[root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Enter PEM pass phrase:

When i entered the password/passphrase it gives invalid configuration file

[root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg
Enter PEM pass phrase:
[WARNING] 318/081440 (2708462) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Configuration file is valid

Kindly suggest what wrong i m doing ?

- Vijay M

[#3]

@VijayM,

The configuration you are using is not correct as it is a mix of pass-through and termination.

You can remove everything from ssl onward in line:

bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem

so it becomes:

bind *:10001

I looked back at my first post and it appears I made a mistake when pasting and forgot to remove the "ssl" part from my pass-through example.  Sorry for the confusion.

NOTE:

If you are doing TLS termination, then being prompted for the key password is expected if you have a key file that is password protected.

NOTE2:

In order to get rid of that WARNING if you want to use termination, add tune.ssl.default-dh-param 2048 to the "global" section of your haproxy.cfg and restart.

In ordre to debug the javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0) issue we really need to see why the handshake is being terminated.  If the log of haproxy and both hiveserver2 servers don't show any TLS messages at the time of the failure, then the next best thing is to do a packet capture on port host where the beeline is run and also on the HiveServer2 server.

Since the TLS handshake is done in the clear, a packet capture can be opened in WireShark where the handshake will be evident.

For example:

1 - runon the beeline host:

# tcpdump -i any -w ~/beeline.pcap port 10001

2 - run on the HiveServer2 host (shut down one so that the load balancer must choose one and you know which):

# tcpdump -i any -w ~/hs2.pcap port 10000

3 - run the beeline command so that it fails

4 - Ctrl-c both tcpdumps

5 - open the pcap files in Wireshark.  You may need to use "decode as..." to decode the 10001 and 10000 ports as SSL/TLS in order to see the TLS handshake.

If you are unfamiliar with packet capture/wireshark, then try this:

# openssl s_client -connect <load_balancer_host>:10001 -msg -debug

This will have openssl client print out the handshake process via the load balancer.

[#4]

Hello bgooley,

Sure, let me try what u suggested.

I would like to use Ssl pass through in haproxy config.

Kindly provide exact configuration needs to define in haproxy config.

- Vijay M

[#5]

@bgooley,

Kindly find below updated hzproxy configuration.

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
backend hiveserver2
mode tcp
balance source
#option ssl-hello-chk
server hiveserver2_1 abc:10000
server hiveserver2_2 xyz:10000

Updated hiveserver2 configuration with hive load balancer.

HiveServer2 Load Balancer  -  abc:10001

Tried to connect through beeline but it still gives the same TLS error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

Tried to test TLS connectivity with openssl command which you suggested. below snap  for the same. Kindly check and suggest.

[root@abc ~]# openssl s_client -connect abc:10001 -msg -debug
CONNECTED(00000003)
write to 0x14e0e00 [0x14f73b0] (289 bytes => 289 (0x121))
0000 - 16 03 01 01 1c 01 00 01-18 03 03 f2 83 c8 e8 a0 ................
0010 - 24 eb da e4 31 75 05 76-62 91 34 b1 4b cc ce 21 $...1u.vb.4.K..!
0020 - 15 99 d7 cc 76 93 78 7d-5e bb 8f 00 00 ac c0 30 ....v.x}^......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........
00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................
00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C..
00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................
00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... ....
0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................
0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................
0120 - 01 .
>>> TLS 1.2 [length 0005]
16 03 01 01 1c
>>> TLS 1.2 Handshake [length 011c], ClientHello
01 00 01 18 03 03 f2 83 c8 e8 a0 24 eb da e4 31
75 05 76 62 91 34 b1 4b cc ce 21 15 99 d7 cc 76
93 78 7d 5e bb 8f 00 00 ac c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00
6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00
87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00
40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00
99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0
2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00
96 00 41 c0 12 c0 08 00 16 00 13 00 10 00 0d c0
0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00
05 00 04 00 ff 01 00 00 43 00 0b 00 04 03 00 01
02 00 0a 00 0a 00 08 00 17 00 19 00 18 00 16 00
23 00 00 00 0d 00 20 00 1e 06 01 06 02 06 03 05
01 05 02 05 03 04 01 04 02 04 03 03 01 03 02 03
03 02 01 02 02 02 03 00 0f 00 01 01
read from 0x14e0e00 [0x14fc910] (7 bytes => 0 (0x0))
140683941742480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1542369743
Timeout : 300 (sec)
Verify return code: 0 (ok)

Tried to shutdown 1 of hiveserver2 instance and tested with beeline but still same issue.

Hiveserver2 TLS certificates are in JKS format.

Kindly suggest.

- Vijay M

[#6]

@VijayM,

The openssl debug information indicates that the client makes a connection to a server but the server does not return a certificate.  Since a direct connection to HiveServer2 does not have the problem, I conclude that your haproxy is still using termination even though your configuration snippet would indicate otherwise.  Based on what you have provided it appears:

1. your connection to port 10001 is using TLS termination at the haproxy

2. the server certificate is not valid so no TLS handshake can be performed.

Basically, the configuration you show cannot be the one that is being used for haproxy that is running and listening on port 10001 so perhaps it was not restarted.

openssl s_client will return the following error if the port it connects to is not listening on TLS:

139972358285128:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:769:

Since you are seeing:

SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177

That indicates there was an actual problem on the server side.  The server in this case must be your haproxy.

So, I think it would be good to list the full haproxy configuration file and also make sure that it really did restart since your last change.  I used your config file and pass-through TLS worked perfectly to my HS2 servers.

I think we must be fighting an haproxy config/restart issue since the frontend/backend you showed last worked for me. I actually copied and pasted your config and changed the hostnames only.

[#7]

Hello bgooley,

I will going to office on Tuesday now, will do the troubleshooting on that
day and if still the same issue can provide you full haproxy config file.

If u can provide ur full haproxy config file then I will compare and will
correct the things.

- Vijay M

[#8]

global


   log 127.0.0.1 local2

   pidfile /var/run/haproxy.pid
   maxconn 4000
   user haproxy
   group haproxy
   daemon
   stats socket /tmp/haproxy
   tune.ssl.default-dh-param 2048

defaults
   mode http
   log global
   option httplog
   option dontlognull
   option forwardfor except 127.0.0.0/8
   option redispatch
   retries 3
   timeout http-request 10s
   timeout queue 1m
   timeout connect 10s
   timeout client 10m
   timeout server 10m
   timeout check 10s
   maxconn 3000

listen admin
   bind *:8000
   stats enable

frontend hiveserver2_front
   bind *:10001
   option tcplog
   mode tcp
   default_backend hiveserver2

backend hiveserver2
   mode tcp
   balance source
   server hs2_1 host1.example.com:10000
   server hs2_2 host2.example.com:10000

[#9]

@bgooley

I have updated haproxy configuration as you provided. Below snap for the same.

global

log 127.0.0.1 local2

pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# stats socket /tmp/haproxy
# tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
# option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout check 10s
maxconn 3000

listen admin
bind *:8000
stats enable

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance source
server hs2_1 abc:10000
server hs2_2 xyz:10000

restarted haproxy and it started with current timestamp. Check and verified.

Below find configuration info

haproxy service and 1st hiveserver instance running on server : abc
2nd hiveserver instance running on server : xyz

Scenario 1:

Both hiveserver2 instance up and running.
Connecting from beeline from server abc, throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)

Connecting from beeline from server xyz is successful, Below snap for the same.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://abc:1> show databases;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://abc:1> show databases;
INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds
INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds
INFO : OK
+----------------+--+
| database_name |
+----------------+--+
| default |
| test1 |
| test |
+----------------+--+

Scenario 2:

hiveserver2 instance stopped on server xyz

Connecting from beeline from server abc, throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)

Connecting from beeline from server xyz is successful, Below snap for the same.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
scan complete in 2ms
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://abc:1> show databases;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connected to: Apache Hive (version 1.1.0-cdh5.15.1)
Driver: Hive JDBC (version 1.1.0-cdh5.15.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
1: jdbc:hive2://abc:1> show databases;
INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds
INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds
INFO : OK
+----------------+--+
| database_name |
+----------------+--+
| default |
| test1 |
| test |
+----------------+--+

Scenario 3:

hiveserver2 instance stopped on server abc

Connecting from beeline from server abc, throws below error.

beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)

Connecting from beeline from server xyz is failed, Below snap for the same. Request from haproxy seems not getting forwarded to xyzs server when abc is not available.

beeline> !connect jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
scan complete in 1ms
Connecting to jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)

Kindly check for all above scenario and suggest.

- VIjay M

[#10]

@bgooley

Did some more troubleshooting. 

Updated alogrithm for load balance to roundrobin from source as mentioned below in haproxy configuration and started to get errors in hiveserver2 logs.

frontend hiveserver2_front
bind *:10001
option tcplog
mode tcp
default_backend hiveserver2

backend hiveserver2
mode tcp
balance roundrobin
server hs2_1 a301-8883-0447.gdzd.ubs.net:10000 check
server hs2_2 a301-8883-2675.gdzd.ubs.net:10000 check

When tried to connect from beeline from any server my 2nd connection always to server xyz and connection getting failed and hiveserver2 logs of xyz server throws below error.

2018-11-19 08:48:05,964 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-44]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:794)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:791)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1904)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:791)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 16 more

Kindly suggest ?

- Vijay M

[#11 · Accepted solution]

> java.security.cert.CertificateException: No subject alternative DNS name matching abc found. 

Hi,

This error is important to note, as it would appear to mean that a certificate is now vailable to the client. The balancing algorithim really has no bearing on this particular issue and you must address this issue. By RFC standard if you use Subject Alt Names (SAN) and a CN the very first entry in the DNS Alt Name field must be the CN of the certificate. The error tells us that abc is not the first entry in DNS Alt Names (SAN).

You need to review the CN and Subject/DNS Alt Names on your certificates in use by Hiveserver 2.