Multiple spark Spark jobs failed

Albap — Sun, 19 Feb 2023 21:17:47 GMT

Spark jobs failed after running for ~ 600hrs with the below error:

ERROR util.Utils: Uncaught exception in thread shutdown-hook-0

java.lang.InterruptedException

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2067)

at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)

at org.apache.spark.streaming.scheduler.JobScheduler.stop(JobScheduler.scala:133)

at org.apache.spark.streaming.StreamingContext$$anonfun$stop$1.apply$mcV$sp(StreamingContext.scala:686)

at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1314)

at org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:685)

at org.apache.spark.streaming.StreamingContext.org$apache$spark$streaming$StreamingContext$$stopOnShutdown(StreamingContext.scala:722)

at org.apache.spark.streaming.StreamingContext$$anonfun$start$1.apply$mcV$sp(StreamingContext.scala:604)

at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)

at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)

at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)

at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1919)

at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)

at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)

at scala.util.Try$.apply(Try.scala:192)

at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)

at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Re: Multiple spark Spark jobs failed

DianaTorres — Thu, 23 Feb 2023 21:05:42 GMT

Hello @Albap , Based on log is difficult to fix the issue, we recommend for you to create a support case, thanks!

Re: Multiple spark Spark jobs failed

RangaReddy — Thu, 30 Mar 2023 09:51:40 GMT

Hi @Albap

Based on the logs, i can see you have created streaming application. By default streaming application will run 24*7, it will stop only when we kill or some interrupted event happen at the system level.

Better way to kill/shutdown the spark streaming applications is by using graceful shutdown.

If you need further help, please raise an cloudera case we will work on.

question Re: Multiple spark Spark jobs failed in Support Questions

Multiple spark Spark jobs failed

Re: Multiple spark Spark jobs failed

Re: Multiple spark Spark jobs failed