Reply
Explorer
Posts: 46
Registered: ‎09-29-2016

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

I get below error sometimes while running Impala query  through JDBC connection using Hive jars:

 

java.sql.SQLException: Error while cleaning up the server resources

                at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:580)

                at gxs.core.hadoop$with_impala_connection_STAR_.invoke(Unknown Source)

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

                at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)

                at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)

                at org.apache.hive.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:173)

                at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:165)

                at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:578)

                ... 25 more

Caused by: java.net.SocketException: Broken pipe

                at java.net.SocketOutputStream.socketWrite0(Native Method)

                at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

                at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

                at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)

                ... 29 more

 

How to resolve this ?

Cloudera Employee
Posts: 6
Registered: ‎08-21-2017

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Is the impalad that was the destination of the connection still up/healthy?

Explorer
Posts: 46
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

yes,but still get this problem many times.

Cloudera Employee
Posts: 28
Registered: ‎12-11-2015

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Is there any loadbalancer inbetween the client and impalad? if yes, try to increase the conntimeout on the LB

Explorer
Posts: 46
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

yes i am using haproxy where i have configured my all impalad nodes and this url is used by jdbc connection.

Cloudera Employee
Posts: 28
Registered: ‎12-11-2015

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Can you paste the haproxy.cfg here. Just want to have a look at the connection timeout configured

 

The error you see happens if the connection between client and impalad gets broken. having a lower connection timeout on haproxy can potentially cause this

Explorer
Posts: 46
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

just a correction, we are using haproxy in UAT env but for PROD we are using  VIP which is created by infrastructure team. here is the UAT config:

 

bash-4.1$ vi haproxy-cdh.cfg
user impala
group impala
daemon

# turn on stats unix socket
#stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#
# You might need to adjust timing values to prevent timeouts.
#---------------------------------------------------------------------
defaults
# mode http
# option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
maxconn 1000
timeout connect 300000
timeout client 300000
timeout server 300000

#
# This sets up the admin page for HA Proxy at port 25002.
#
listen stats :25002
balance
mode http
stats enable
stats auth username:password

# This is the setup for Impala. Impala client connect to load_balancer_host:25003.
# HAProxy will balance connections among the list of servers listed below.
# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver.
# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
#listen impala :25053
listen impala :25003
timeout client 3600000
timeout server 3600000
balance leastconn

 

Cloudera Employee
Posts: 28
Registered: ‎12-11-2015

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Is the issue reported in this thread happened in UAT or in PROD?

 

From UAT configs, the default timeouts are around 5 minutes and overridden timeouts are around 1 hr.

 

Is the error that you posted happens for long running jdbc application maintaining a single connection ? Can you check if you increase the timeouts[lets say 2hr] further, the issue still happens?

Explorer
Posts: 46
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

the reported issue is in PROD though we have got the error in UAT too but very rare. we are having 15 JDBC  connection  in connection pool. 

Announcements