Support Questions
Find answers, ask questions, and share your expertise

Running HiveServer2 in HTTP and Binary mode (on different hosts) : Intermittent failure in connecting to HiveServer2

Running HiveServer2 in HTTP and Binary mode (on different hosts) : Intermittent failure in connecting to HiveServer2

New Contributor

Hello Team,

 

I have HiveServer2 running in Binary (port: 10000) as well as HTTP mode (port: 10001), on different hosts.

While connecting to hive cli, it sometimes connects to the hiveserver and sometimes throws error (attached below). I have not changed anything on the configuration files on Hive, and this seems like a strange behaviour.

 

Error:

 

 

ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

 

 

 

Logs:

 

 

$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Connecting to jdbc:hive2://master1:2181,master2:2181,master3:2181/default;httpPath=cliservice;password=PASSWORD;principal=hive/_HOST@REALM;serviceDiscoveryMode=zooKeeper;transportMode=http;user=USERNAME;zooKeeperNamespace=hiveserver2

 19:42:58 [main]: INFO jdbc.HiveConnection: Connected to master3:10000
 19:42:58 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:853) [hive-jdbc-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:316) [hive-jdbc-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_112]
        at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_112]
        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.Commands.connect(Commands.java:1643) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.Commands.connect(Commands.java:1538) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
        at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1456) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1495) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.defaultBeelineConnect(BeeLine.java:1154) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:808) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1108) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
        at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.3.1.4.0-315.jar:?]
        at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.3.1.4.0-315.jar:?]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[?:1.8.0_112]
        at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_112]
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.10.jar:4.4.10]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        ... 33 more
 19:42:58 [main]: WARN jdbc.HiveConnection: Failed to connect to master3:10000
 19:42:58 [main]: WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri: jdbc:hive2://master3:10000/default;httpPath=cliservice;password=PASSWORD;principal=hive/_HOST@REALM;serviceDiscoveryMode=zooKeeper;transportMode=http;user=USERNAME;zooKeeperNamespace=hiveserver2: Could not establish connection to jdbc:hive2://master3:10000/default;httpPath=cliservice;password=PASSWORD;principal=hive/_HOST@REALM;serviceDiscoveryMode=zooKeeper;transportMode=http;user=USERNAME;zooKeeperNamespace=hiveserver2: java.net.SocketException: Connection reset Retrying 0 of 1
 19:42:58 [main]: INFO jdbc.HiveConnection: Connected to edgenode.REALM:10000
  [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: org.apache.http.conn.HttpHostConnectException: Connect to edgenode.REALM:10000 [edgenode.REALM/172.40.40.40] failed: Connection refused (Connection refused)
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:853) [hive-jdbc-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:316) [hive-jdbc-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_112]
        at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_112]
        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.Commands.connect(Commands.java:1643) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.Commands.connect(Commands.java:1538) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
        at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1456) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1495) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.defaultBeelineConnect(BeeLine.java:1154) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:808) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1108) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528) [hive-beeline-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
        at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.3.1.4.0-315.jar:?]
        at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.3.1.4.0-315.jar:?]
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to edgenode.REALM:10000 [edgenode.REALM/172.40.40.40] failed: Connection refused (Connection refused)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        ... 33 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_112]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_112]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_112]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_112]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_112]
        at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_112]
        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar:4.5.6]
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
        ... 33 more
  [main]: WARN jdbc.HiveConnection: Failed to connect to edgenode.REALM:10000
  [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Could not establish connection to jdbc:hive2://edgenode.REALM:10000/default;httpPath=cliservice;password=PASSWORD;principal=hive/_HOST@REALM;serviceDiscoveryMode=zooKeeper;transportMode=http;user=USERNAME;zooKeeperNamespace=hiveserver2: org.apache.http.conn.HttpHostConnectException: Connect to edgenode.REALM:10000 [edgenode.REALM/172.40.40.40] failed: Connection refused (Connection refused) (state=08S01,code=0)
Beeline version 3.1.0.3.1.4.0-315 by Apache Hive
beeline>

 

 

     

 

Don't have an account?