question SparkContext was shut down in Support Questions

SparkContext was shut down

poorva — Fri, 16 Sep 2022 02:07:23 GMT

how to know why sparkcontext was shut down ??ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.spark.SparkException: Job 1 cancelled because SparkContext was shut down

Re: SparkContext was shut down

RangaReddy — Fri, 16 Sep 2022 12:19:10 GMT

Hi @poorva

Please check the application logs for failed applications from Resource Manager UI, there exception message is printed. Fix the exception and resubmit the job.

Re: SparkContext was shut down

vaishaakb — Mon, 19 Sep 2022 16:29:52 GMT

Hello @poorva Did you get a chance to review the Failed Application logs from Resource Manager UI?

You can also share the exception from the same time frame so we can help you further with a plan of action.

Let us know your thoughts.

Re: SparkContext was shut down

VidyaSargur — Mon, 26 Sep 2022 11:29:01 GMT

@poorva, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.