Support Questions

Find answers, ask questions, and share your expertise

Cloudera Hive Broke Pipe Error

avatar
New Contributor

I got the below error when running a cloudera cluster that connects to a hive database and pulls data.
If I restart the Cloudera service, it works again, but it's not good for production.
Please let me know if you know a solution for this issue.

 

 

scala.collection.parallel.CompositeThrowable: Multiple exceptions thrown during a parallel computation: java.sql.SQLException: org.apache.thrift.transport.TTransportException
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:309)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate.jsk$JskAggregate$$createJskResultTbl(JskAggregate.scala:223)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:176)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2.apply(JskAggregate.scala:127)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2.apply(JskAggregate.scala:82)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:309)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:309)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
common.QueryExecutor.execQuery(QueryExecutor.scala:68)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)

 

 

 

4 REPLIES 4

avatar
Community Manager

@keifuku, Welcome to our community! To help you get the best possible answer, I have tagged in our Hive experts @smruti @Shelton @Asok @marudhu_ramaiah who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Master Collaborator

@keifuku  Broken Pipe means the socket connection to HS2 is getting terminated either from Client or Server end. Do you have any timeout set in Client end, because we do not have any timeout configured in Hiveserver2. Also do check if your HS2 servers are under heavy load. You may get an idea by looking at number of active sessions, and heap utilization. If you are seeing too many connections, or notice JVM pause messages in HS2 logs, consider adding more resources, or adding more HS2 instances(place them behind a load balancer).

avatar
New Contributor

@smruti 
Thanks for your reply.

I would check the Hive timeout settings, HS2 logs, and server's load.

avatar
Community Manager

@keifuku, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: