Knox over Hive SSL Failed

[#1]

Hi,

Before enabling SSL over Hive I was able to access Hive through Knox. After enabling SSL over Hive I am not able to access Hive through Knox.

Getting following error in beeline:

Beeline version 1.2.1.2.3.4.0-3485 by Apache Hive
beeline> !connect 'jdbc:hive2://hadmgrndcc03-1.test.org:8443/default/;ssl=true;sslTrustStore=/var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks;trustStorePassword=test123;transportMode=http;httpPath=gateway/default/hive'
Connecting to jdbc:hive2://hadmgrndcc03-1.test.org:8443/default/;ssl=true;sslTrustStore=/var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks;trustStorePassword=test123;transportMode=http;httpPath=gateway/default/hive
Enter username for jdbc:hive2://hadmgrndcc03-1.test.org:8443/default/;ssl=true;sslTrustStore=/var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks;trustStorePassword=test123;transportMode=http;httpPath=gateway/default/hive: guest
Enter password for jdbc:hive2://hadmgrndcc03-1.test.org:8443/default/;ssl=true;sslTrustStore=/var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks;trustStorePassword=test123;transportMode=http;httpPath=gateway/default/hive: **************
Error: Could not open client transport with JDBC Uri: jdbc:hive2://hadmgrndcc03-1.test.org:8443/default/;ssl=true;sslTrustStore=/var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks;trustStorePassword=test123;transportMode=http;httpPath=gateway/default/hive: Could not create http connection to jdbc:hive2://hadmgrndcc03-1.test.org:8443/default/;ssl=true;sslTrustStore=/var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks;trustStorePassword=test123;transportMode=http;httpPath=gateway/default/hive. HTTP Response code: 500 (state=08S01,code=0)
0: jdbc:hive2://hadmgrndcc03-1.test.org:84 (closed)>

Also, Getting following error in Knox gateway log:

2018-05-08 08:32:12,279 INFO hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true 2018-05-08 08:32:12,737 WARN hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(129)) - Connection exception dispatching request: http://hadmgrndcc03-3.test.org:10001/cliservice?user.name=guest org.apache.http.NoHttpResponseException: hadmgrndcc03-3.test.org:10001 failed to respond org.apache.http.NoHttpResponseException: hadmgrndcc03-3.test.org:10001 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153)


2018-05-08 08:32:12,767 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(63)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2018-05-08 08:32:12,767 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(63)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2018-05-08 08:32:12,768 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(66)) - Failed to execute filter: javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: Service connectivity error.
2018-05-08 08:32:12,776 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(66)) - Failed to execute filter: javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: Service connectivity error.
2018-05-08 08:32:12,776 ERROR hadoop.gateway (GatewayFilter.java:doFilter(135)) - Gateway processing failed: javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: Service connectivity error.
javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: Service connectivity error.
        at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)

How to resolve it? Please suggest.

Thanks in Advance.

[#2]

@Bhushan Kandalkar did you add the hive certificate to the knox host cacerts and restart Knox? This may help resolve the problem.

#open console to knox host

# run the following command to locate the jdk used by knox
ps -ef | grep -i knox

# run the following command to import the hive certificate to the default cacerts truststore
keytool -import -file hive.crt -keystore /<knox_jdk_path>/jre/lib/security/cacerts
    -storepass changeit -alias hive

Note: if you add any comments to this post please make sure you tag my name. Also If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#3]

@Felix Albani, @Alex Miller , we have Hiveserver2 HA setup in our environment. I have done following things:

Imported both Hiveservers certificates in Knox gateway.jks file:

echo -n | openssl s_client -connect hadmgrndcc03-2.test.org:10001 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > hadmgrndcc03-2.test.org.pem
echo -n | openssl s_client -connect hadmgrndcc03-3.test.org:10001 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > hadmgrndcc03-3.test.org.pem

keytool -import -alias hadmgrndcc03-3.test.org -file hadmgrndcc03-3.test.orgt.pem -keystore /var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks
keytool -import -alias hadmgrndcc03-2.test.org -file hadmgrndcc03-2.test.orgt.pem -keystore /var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks

Imported both hiveserver2 certificates on cacert file of Knox machine

keytool -import -alias hadmgrndcc03-3.test.org -file hadmgrndcc03-3.test.org.pem -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
keytool -import -alias hadmgrndcc03-2.test.org -file hadmgrndcc03-2.test.org.pem -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts


keytool -import -alias hadmgrndcc03-3.test.org -file hadmgrndcc03-3.test.org.pem -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
keytool -import -alias hadmgrndcc03-2.test.org -file hadmgrndcc03-2.test.org.pem -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

On both Hiveservers imported Knox certificate to cacert:

echo -n | openssl s_client -connect hadmgrndcc03-1.test.org:8443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > knox.pem

keytool -import -alias knox -file knox.pem -keystore /usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts

Still getting same error. How to resolve it?

Please suggest.

[#4]

@Bhushan Kandalkar above steps look good to me. Do you see any errors on hiveserver2.log?

[#5]

@Felix Albani, not getting any errors in Hiveserver2 logs:

2018-05-08 11:26:42,453 INFO  [main-SendThread(hadmgrndcc03-3.lifeway.org:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1279)) - Session establishment complete on server hadmgrndcc03-3.lifeway.org/172.17.20.33:2181, sessionid = 0x3632acb1c590011, negotiated timeout = 40000
2018-05-08 11:26:42,461 INFO  [main-EventThread]: state.ConnectionStateManager (ConnectionStateManager.java:postState(228)) - State change: CONNECTED
2018-05-08 11:26:42,527 INFO  [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(234)) - Created a znode on ZooKeeper for HiveServer2 uri: hadmgrndcc03-3.lifeway.org:10001
2018-05-08 11:26:42,701 INFO  [Thread-9]: server.Server (Server.java:doStart(252)) - jetty-7.6.0.v20120127
2018-05-08 11:26:42,753 INFO  [Thread-9]: handler.ContextHandler (ContextHandler.java:startContext(737)) - started o.e.j.s.ServletContextHandler{/,null}
2018-05-08 11:26:43,080 INFO  [Thread-9]: ssl.SslContextFactory (SslContextFactory.java:doStart(297)) - Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
2018-05-08 11:26:43,120 INFO  [Thread-9]: server.AbstractConnector (AbstractConnector.java:doStart(333)) - Started SslSelectChannelConnector@0.0.0.0:10001
2018-05-08 11:26:43,120 INFO  [Thread-9]: thrift.ThriftCLIService (ThriftHttpCLIService.java:run(141)) - Started ThriftHttpCLIService in https mode on port 10001 path=/cliservice/* with 5...500 worker threads

[#6]

@Vipin Rathor @Ancil McBarnett @Kevin Minder.. Any suggestions please.

[#7]

Hello @Bhushan Kandalkar,

At this point, I'd enable debug for Beeline and check where exactly it is failing. Also, I'm surprised to see that both HS2 are not showing any sign of error whereas Beeline is showing '500 internal server error'. I hope you have checked both the HS2 logs.

Anyways, Beeline debug should tell us more.

Hope this helps!

UPDATE:
I looked at it again and that '500 internal server error' is actually from Knox and due to this line:

2018-05-0808:32:12,767 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(63))-Failed to execute filter: java.io.IOException:Service connectivity error.

This tells me that Knox is not able to connect your authentication server (defined in topology). So instead of debug in Beeline, I'd enable debug in Knox to know more.

Also, are you able to make an HDFS call via Knox using the same topology (just to verify topology configuration).

[#8]

Thanks @Vipin Rathor

[#9 · Accepted solution]

@Bhushan Kandalkar When I looked at your original error from the knox gateway.log I see:

dispatching request: http://hadmgrndcc03-3.test.org:10001/cliservice?user.name=guest org.apache.http.NoHttpResponseException:

The gateway-audit.log should show this as well upon dispatch there is a problem knox communicating to hive.

This tells me that you never changed your KNOX topology to include the hive service links with the correct protocol from http to https. Make sure knox knows that you should be using https vs http in the topology to communicate to hive.

[#10]

Thanks @dvillarreal.