Cloudera Hive Broke Pipe Error

[#1]

I got the below error when running a cloudera cluster that connects to a hive database and pulls data.
If I restart the Cloudera service, it works again, but it's not good for production.
Please let me know if you know a solution for this issue.

scala.collection.parallel.CompositeThrowable: Multiple exceptions thrown during a parallel computation: java.sql.SQLException: org.apache.thrift.transport.TTransportException
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:309)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate.jsk$JskAggregate$$createJskResultTbl(JskAggregate.scala:223)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:176)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2.apply(JskAggregate.scala:127)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2.apply(JskAggregate.scala:82)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:309)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:309)
common.QueryExecutor.execQuery(QueryExecutor.scala:79)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)
.
.
.

java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
common.QueryExecutor.execQuery(QueryExecutor.scala:68)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:210)
jsk.JskAggregate$$anonfun$jsk$JskAggregate$$insertJskResultTmpTbl$1.apply(JskAggregate.scala:202)
scala.collection.immutable.List.foreach(List.scala:318)
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
jsk.JskAggregate.jsk$JskAggregate$$insertJskResultTmpTbl(JskAggregate.scala:202)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:161)
jsk.JskAggregate$$anonfun$main$2$$anonfun$apply$2$$anonfun$apply$8.apply(JskAggregate.scala:127)

[#2]

@keifuku, Welcome to our community! To help you get the best possible answer, I have tagged in our Hive experts @smruti @Shelton @Asok @marudhu_ramaiah who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@keifuku  Broken Pipe means the socket connection to HS2 is getting terminated either from Client or Server end. Do you have any timeout set in Client end, because we do not have any timeout configured in Hiveserver2. Also do check if your HS2 servers are under heavy load. You may get an idea by looking at number of active sessions, and heap utilization. If you are seeing too many connections, or notice JVM pause messages in HS2 logs, consider adding more resources, or adding more HS2 instances(place them behind a load balancer).

[#4]

@smruti 
Thanks for your reply.

I would check the Hive timeout settings, HS2 logs, and server's load.

[#5]

@keifuku, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.