Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SPARK-13566: [Spark-CORE] Avoid deadlock between BlockManager and Executor Thread

SPARK-13566: [Spark-CORE] Avoid deadlock between BlockManager and Executor Thread

New Contributor



We encountered a dead-lock error in the BlockManager many times described in the issue: It should be fixed in the Spark version 1.6.2. Is this fix already included (or planned) in CDH? We couldn't find this issue in the CDH release notes. We're using 'cdh5.9.0.p0.23'.


Exception e.g.:



Caused by: org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [1200 seconds]. This timeout is controlled by at$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) at at org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1623) at org.apache.spark.rdd.RDD.unpersist(RDD.scala:203)



Re: SPARK-13566: [Spark-CORE] Avoid deadlock between BlockManager and Executor Thread

Master Collaborator

Good question, you can always find exactly what's in a release by looking in the branches or consulting for example ...


You are right, it isn't in the 5.9 release, and I would have generally expected it to be. Open a support case to ask for it to be cherry-picked, if you're able to do that.

Don't have an account?
Coming from Hortonworks? Activate your account here