beeline/JDBC/ODBC session timeout / connection failure

[#1]

We are querying HS2 using knox through beeline and also other jdbc tool and getting frequent disconnection. Below is the url for connection thorugh beeline:-

jdbc:hive2://c3master03-nn.abc.org:8445/;ssl=true?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive

After connection if i do not query for a minute then i got below error(Same with Squirrel JDBC client as well)

Getting log thread is interrupted, since query is done!
Error: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond (state=08S01,code=0)
java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond
        at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:305)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:238)
        at org.apache.hive.beeline.Commands.execute(Commands.java:863)
        at org.apache.hive.beeline.Commands.sql(Commands.java:728)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:993)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:833)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:791)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:491)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:474)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1363)
        at com.sun.proxy.$Proxy0.ExecuteStatement(Unknown Source)
        at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:296)
        ... 14 more
Caused by: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
        ... 26 more

Even After this exception , if i rerun the query on same beeline window , it will get executed and show me the result.

And after the wait for 1 minute if i execute the same query or any other query , same exception for one time and on rerun result is there...What is this weird behavior..

Even the below properties have sufficient values.

1. hive.server2.session.check.interval
2. hive.server2.idle.operation.timeout
3. hive.server2.idle.session.timeout

Can some one help what is the issue or configuration changes required...

[#2]

@Anwaar Siddiqui

It seems to be a knox bug https://issues.apache.org/jira/browse/KNOX-890

Workaround:-

Append "http.header.Connection=close" in the JDBC connection string.

For example, with Beeline, use the following command:

beeline -u "jdbc:hive2://sandbox.hortonworks.com:8443/ ;ssl=false;sslTrustStore=/tmp/myNewTrustStore.jks;trustStorePassword=changeit;transportMode =http;httpPath=gateway/default/hive;http.header.Connection=close" -n admin -p admin-password

[#3]

@Rajesh...

Thanks ...It is working for beeline.

Since it is a bug in Knox , can we upgrade from KNOX 0.9 to Knox 0.12 for HDP2.5.3 ? is there any document for that as i was not able to find any doc for upgrading knox ?

[#4]

@Anwaar Siddiqui Great it works for you. Please accept the answer.

You can't just upgrade one component in the stack. You have to consider moving to HDP-2.6.X latest version which has Knox 0.12. Hope this helps.