Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

avatar

I get below error sometimes while running Impala query  through JDBC connection using Hive jars:

 

java.sql.SQLException: Error while cleaning up the server resources

                at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:580)

                at gxs.core.hadoop$with_impala_connection_STAR_.invoke(Unknown Source)

Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe

                at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)

                at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)

                at org.apache.hive.service.cli.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:173)

                at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:165)

                at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:578)

                ... 25 more

Caused by: java.net.SocketException: Broken pipe

                at java.net.SocketOutputStream.socketWrite0(Native Method)

                at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

                at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

                at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)

                ... 29 more

 

How to resolve this ?

9 REPLIES 9

avatar
Cloudera Employee

Is the impalad that was the destination of the connection still up/healthy?

avatar

yes,but still get this problem many times.

avatar
Master Collaborator

Is there any loadbalancer inbetween the client and impalad? if yes, try to increase the conntimeout on the LB

avatar

yes i am using haproxy where i have configured my all impalad nodes and this url is used by jdbc connection.

avatar
Master Collaborator

Can you paste the haproxy.cfg here. Just want to have a look at the connection timeout configured

 

The error you see happens if the connection between client and impalad gets broken. having a lower connection timeout on haproxy can potentially cause this

avatar

just a correction, we are using haproxy in UAT env but for PROD we are using  VIP which is created by infrastructure team. here is the UAT config:

 

bash-4.1$ vi haproxy-cdh.cfg
user impala
group impala
daemon

# turn on stats unix socket
#stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#
# You might need to adjust timing values to prevent timeouts.
#---------------------------------------------------------------------
defaults
# mode http
# option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
maxconn 1000
timeout connect 300000
timeout client 300000
timeout server 300000

#
# This sets up the admin page for HA Proxy at port 25002.
#
listen stats :25002
balance
mode http
stats enable
stats auth username:password

# This is the setup for Impala. Impala client connect to load_balancer_host:25003.
# HAProxy will balance connections among the list of servers listed below.
# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver.
# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
#listen impala :25053
listen impala :25003
timeout client 3600000
timeout server 3600000
balance leastconn

 

avatar
Master Collaborator

Is the issue reported in this thread happened in UAT or in PROD?

 

From UAT configs, the default timeouts are around 5 minutes and overridden timeouts are around 1 hr.

 

Is the error that you posted happens for long running jdbc application maintaining a single connection ? Can you check if you increase the timeouts[lets say 2hr] further, the issue still happens?

avatar

the reported issue is in PROD though we have got the error in UAT too but very rare. we are having 15 JDBC  connection  in connection pool. 

avatar
New Contributor
I am having the same issue..using c3p0 pool if i retry it works any sample code of connection pooling