CDH 5.7.0 Spark 1.6.0 Has numerous RejectedExecutionException messages

BillM. — Fri, 16 Sep 2022 10:38:19 GMT

I have an existing job which has run successfully many times in the past but a recent launch using Yarn Cluster Mode with the same configuration as an earlier atttempt (which failed for other reasons) generated many instances of

6308860 [Stage 1:==============================================> (12508 + 712) / 13220]java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@6d3616e6 rejected from java.util.concurrent.ThreadPoolExecutor@4eef216[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11 70]
6308861 at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
6308862 at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
6308863 at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
6308864 at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:122)
6308865 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308866 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308867 at scala.concurrent.Promise$class.complete(Promise.scala:55)
6308868 at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
6308869 at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
6308870 at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
6308871 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
6308872 at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
6308873 at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
6308874 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308875 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308876 at scala.concurrent.Promise$class.complete(Promise.scala:55)
6308877 at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
6308878 at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
6308879 at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
6308880 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
6308881 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:643)
6308882 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:658)
6308883 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
6308884 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
6308885 at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
6308886 at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:634)
6308887 at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
6308888 at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:685)
6308889 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308890 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308891 at scala.concurrent.Promise$class.trySuccess(Promise.scala:94)
6308892 at scala.concurrent.impl.Promise$DefaultPromise.trySuccess(Promise.scala:153)
6308893 at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onSuccess$1(NettyRpcEnv.scala:215)
6308894 at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$ask$2.apply(NettyRpcEnv.scala:223)
6308895 at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$ask$2.apply(NettyRpcEnv.scala:222)
6308896 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
6308897 at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
6308898 at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
6308899 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308900 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308901 at scala.concurrent.Promise$class.complete(Promise.scala:55)
6308902 at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
6308903 at scala.concurrent.Promise$class.success(Promise.scala:86)
6308904 at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
6308905 at org.apache.spark.rpc.netty.LocalNettyRpcCallContext.send(NettyRpcCallContext.scala:50)
6308906 at org.apache.spark.rpc.netty.NettyRpcCallContext.reply(NettyRpcCallContext.scala:32)
6308907 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7$$anonfun$apply$8.apply(YarnAllocator.scala:517)
6308908 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7$$anonfun$apply$8.apply(YarnAllocator.scala:517)
6308909 at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
6308910 at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
6308911 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7.apply(YarnAllocator.scala:517)
6308912 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7.apply(YarnAllocator.scala:512)
6308913 at scala.Option.foreach(Option.scala:236)
6308914 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1.apply(YarnAllocator.scala:512)
6308915 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1.apply(YarnAllocator.scala:442)
[Deleted items from the stack, various places in spark and its libraries for brevity]

6308919 at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
6308920 at org.apache.spark.deploy.yarn.YarnAllocator.processCompletedContainers(YarnAllocator.scala:442)
6308921 at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:242)
6308922 at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:368)

This is a new symptom to me, any idea what causes it and whether this is responsible for this job's failure? Is there some reason why these messages are not logged with timestamps?

Re: CDH 5.7.0 Spark 1.6.0 Has numerous RejectedExecutionException messages

Dat Pham — Wed, 14 Sep 2016 21:05:30 GMT

The new error message related to RejectedExecutionException is most likely due to the fact that threads are being submitted to executor that have been lost/killed hence you see these type of messages (a good read on RejectedExecutionException can be found here [1]): === java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@6143b0a3 rejected from java.util.concurrent.ThreadPoolExecutor@4eef216[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1170] === [1] https://examples.javacodegeeks.com/core-java/util/concurrent/rejectedexecutionexception/java-util-concurrent-rejectedexecutionexception-how-to-solve-rejectedexecutionexception

question CDH 5.7.0 Spark 1.6.0 Has numerous RejectedExecutionException messages in Archives of Support Questions (Read Only)

CDH 5.7.0 Spark 1.6.0 Has numerous RejectedExecutionException messages

Re: CDH 5.7.0 Spark 1.6.0 Has numerous RejectedExecutionException messages