CDH 5.7.0 Spark 1.6.0 Has numerous RejectedExecutionException messages

[#1]

I have an existing job which has run successfully many times in the past but a recent launch using Yarn Cluster Mode with the same configuration as an earlier atttempt (which failed for other reasons) generated many instances of 

6308860 [Stage 1:==============================================> (12508 + 712) / 13220]java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@6d3616e6 rejected from java.util.concurrent.ThreadPoolExecutor@4eef216[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 11 70]
6308861 at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
6308862 at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
6308863 at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
6308864 at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:122)
6308865 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308866 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308867 at scala.concurrent.Promise$class.complete(Promise.scala:55)
6308868 at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
6308869 at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
6308870 at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324)
6308871 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
6308872 at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
6308873 at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
6308874 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308875 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308876 at scala.concurrent.Promise$class.complete(Promise.scala:55)
6308877 at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
6308878 at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
6308879 at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
6308880 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
6308881 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.processBatch$1(Future.scala:643)
6308882 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Future.scala:658)
6308883 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
6308884 at scala.concurrent.Future$InternalCallbackExecutor$Batch$$anonfun$run$1.apply(Future.scala:635)
6308885 at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
6308886 at scala.concurrent.Future$InternalCallbackExecutor$Batch.run(Future.scala:634)
6308887 at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
6308888 at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:685)
6308889 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308890 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308891 at scala.concurrent.Promise$class.trySuccess(Promise.scala:94)
6308892 at scala.concurrent.impl.Promise$DefaultPromise.trySuccess(Promise.scala:153)
6308893 at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onSuccess$1(NettyRpcEnv.scala:215)
6308894 at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$ask$2.apply(NettyRpcEnv.scala:223)
6308895 at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$ask$2.apply(NettyRpcEnv.scala:222)
6308896 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
6308897 at org.spark-project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
6308898 at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:133)
6308899 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
6308900 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
6308901 at scala.concurrent.Promise$class.complete(Promise.scala:55)
6308902 at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
6308903 at scala.concurrent.Promise$class.success(Promise.scala:86)
6308904 at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
6308905 at org.apache.spark.rpc.netty.LocalNettyRpcCallContext.send(NettyRpcCallContext.scala:50)
6308906 at org.apache.spark.rpc.netty.NettyRpcCallContext.reply(NettyRpcCallContext.scala:32)
6308907 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7$$anonfun$apply$8.apply(YarnAllocator.scala:517)
6308908 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7$$anonfun$apply$8.apply(YarnAllocator.scala:517)
6308909 at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
6308910 at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
6308911 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7.apply(YarnAllocator.scala:517)
6308912 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1$$anonfun$apply$7.apply(YarnAllocator.scala:512)
6308913 at scala.Option.foreach(Option.scala:236)
6308914 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1.apply(YarnAllocator.scala:512)
6308915 at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$processCompletedContainers$1.apply(YarnAllocator.scala:442)
[Deleted items from the stack, various places in spark and its libraries for brevity]

6308919 at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
6308920 at org.apache.spark.deploy.yarn.YarnAllocator.processCompletedContainers(YarnAllocator.scala:442)
6308921 at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:242)
6308922 at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:368)

This is a new symptom to me, any idea what causes it and whether this is responsible for this job's failure?  Is there some reason why these messages are not logged with timestamps?

[#2 · Accepted solution]

The new error message related to RejectedExecutionException is most likely due to the fact that threads are being submitted to executor that have been lost/killed hence you see these type of messages (a good read on RejectedExecutionException can be found here [1]): === java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@6143b0a3 rejected from java.util.concurrent.ThreadPoolExecutor@4eef216[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1170] === [1] https://examples.javacodegeeks.com/core-java/util/concurrent/rejectedexecutionexception/java-util-co...