Support Questions
Find answers, ask questions, and share your expertise

beeline/JDBC/ODBC session timeout / connection failure

beeline/JDBC/ODBC session timeout / connection failure

Contributor

We are querying HS2 using knox through beeline and also other jdbc tool and getting frequent disconnection. Below is the url for connection thorugh beeline:-

jdbc:hive2://c3master03-nn.abc.org:8445/;ssl=true?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive

After connection if i do not query for a minute then i got below error(Same with Squirrel JDBC client as well)

Getting log thread is interrupted, since query is done!
Error: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond (state=08S01,code=0)
java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond
        at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:305)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:238)
        at org.apache.hive.beeline.Commands.execute(Commands.java:863)
        at org.apache.hive.beeline.Commands.sql(Commands.java:728)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:993)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:833)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:791)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:491)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:474)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1363)
        at com.sun.proxy.$Proxy0.ExecuteStatement(Unknown Source)
        at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:296)
        ... 14 more
Caused by: org.apache.http.NoHttpResponseException: c3master03-nn.abc.org:8445 failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
        ... 26 more

Even After this exception , if i rerun the query on same beeline window , it will get executed and show me the result.

And after the wait for 1 minute if i execute the same query or any other query , same exception for one time and on rerun result is there...What is this weird behavior..

Even the below properties have sufficient values.

  1. hive.server2.session.check.interval
  2. hive.server2.idle.operation.timeout
  3. hive.server2.idle.session.timeout

Can some one help what is the issue or configuration changes required...

3 REPLIES 3

Re: beeline/JDBC/ODBC session timeout / connection failure

Contributor

@Anwaar Siddiqui

It seems to be a knox bug https://issues.apache.org/jira/browse/KNOX-890

Workaround:-

Append "http.header.Connection=close" in the JDBC connection string.

For example, with Beeline, use the following command:

beeline -u "jdbc:hive2://sandbox.hortonworks.com:8443/ ;ssl=false;sslTrustStore=/tmp/myNewTrustStore.jks;trustStorePassword=changeit;transportMode =http;httpPath=gateway/default/hive;http.header.Connection=close" -n admin -p admin-password

Re: beeline/JDBC/ODBC session timeout / connection failure

Contributor

@Rajesh...

Thanks ...It is working for beeline.

Since it is a bug in Knox , can we upgrade from KNOX 0.9 to Knox 0.12 for HDP2.5.3 ? is there any document for that as i was not able to find any doc for upgrading knox ?

Re: beeline/JDBC/ODBC session timeout / connection failure

Contributor

@Anwaar Siddiqui Great it works for you. Please accept the answer.

You can't just upgrade one component in the stack. You have to consider moving to HDP-2.6.X latest version which has Knox 0.12. Hope this helps.