Reply
Explorer
Posts: 54
Registered: ‎09-29-2016

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

I get below error sometimes while running Impala query  through JDBC connection using Hive jars:

 

java.sql.SQLException: Error while cleaning up the server resources

                at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:580)

                at gxs.core.hadoop$with_impala_connection_STAR_.invoke(Unknown Source)

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

                at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)

                at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)

                at org.apache.hive.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:173)

                at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:165)

                at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:578)

                ... 25 more

Caused by: java.net.SocketException: Broken pipe

                at java.net.SocketOutputStream.socketWrite0(Native Method)

                at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

                at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

                at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)

                ... 29 more

 

How to resolve this ?

Cloudera Employee
Posts: 6
Registered: ‎08-21-2017

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Is the impalad that was the destination of the connection still up/healthy?

Explorer
Posts: 54
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

yes,but still get this problem many times.

Cloudera Employee
Posts: 33
Registered: ‎12-11-2015

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Is there any loadbalancer inbetween the client and impalad? if yes, try to increase the conntimeout on the LB

Explorer
Posts: 54
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

yes i am using haproxy where i have configured my all impalad nodes and this url is used by jdbc connection.

Cloudera Employee
Posts: 33
Registered: ‎12-11-2015

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Can you paste the haproxy.cfg here. Just want to have a look at the connection timeout configured

 

The error you see happens if the connection between client and impalad gets broken. having a lower connection timeout on haproxy can potentially cause this

Explorer
Posts: 54
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

just a correction, we are using haproxy in UAT env but for PROD we are using  VIP which is created by infrastructure team. here is the UAT config:

 

bash-4.1$ vi haproxy-cdh.cfg
user impala
group impala
daemon

# turn on stats unix socket
#stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#
# You might need to adjust timing values to prevent timeouts.
#---------------------------------------------------------------------
defaults
# mode http
# option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
maxconn 1000
timeout connect 300000
timeout client 300000
timeout server 300000

#
# This sets up the admin page for HA Proxy at port 25002.
#
listen stats :25002
balance
mode http
stats enable
stats auth username:password

# This is the setup for Impala. Impala client connect to load_balancer_host:25003.
# HAProxy will balance connections among the list of servers listed below.
# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver.
# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
#listen impala :25053
listen impala :25003
timeout client 3600000
timeout server 3600000
balance leastconn

 

Cloudera Employee
Posts: 33
Registered: ‎12-11-2015

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

Is the issue reported in this thread happened in UAT or in PROD?

 

From UAT configs, the default timeouts are around 5 minutes and overridden timeouts are around 1 hr.

 

Is the error that you posted happens for long running jdbc application maintaining a single connection ? Can you check if you increase the timeouts[lets say 2hr] further, the issue still happens?

Highlighted
Explorer
Posts: 54
Registered: ‎09-29-2016

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

the reported issue is in PROD though we have got the error in UAT too but very rare. we are having 15 JDBC  connection  in connection pool. 

New Contributor
Posts: 1
Registered: ‎09-05-2018

Re: Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pip

I am having the same issue..using c3p0 pool if i retry it works any sample code of connection pooling
Announcements