question Re: Some questions with Spark Listener in Support Questions

Some questions with Spark Listener

Meepoljd — Wed, 07 Dec 2022 03:34:01 GMT

Hi,recently I have been trying to track my spark application using SparkListener.

Now I have a problem:

There is a application, the result of its execution is SUCCESSED on Yarn WebUI,but it's actually a failed application.

And my listener class can not get the error message,the application log only show the Start events for failed jobs and tasks.

Some can help me ?

Re: Some questions with Spark Listener

RangaReddy — Wed, 25 Oct 2023 04:54:27 GMT

Hi @Meepoljd

Have you checked any exceptions/warnings in the spark application logs and event logs. And also check you listeners are implemented properly and spark application handled exceptions correctly.

Re: Some questions with Spark Listener

tsokorai — Wed, 01 Nov 2023 15:07:14 GMT

onStageCompleted

you should have also the failures and the info on why it failed with StageInfo.failureReason()

Check on the standard event log JSON if the logger stored that event, if it also does not have it, then it means your listener might have an issue.

If you are not really seeing the StageCompleted event anywhere ( not even on the event log), it probably means the failure was more than just a regular error and it had a major error/crash that aborted the execution of that stage in a way that no StageCompleted was ever sent.

Also bear in mind that there are specific listeners for SQL execution with fail/success events: QueryExecutionListener .

Re: Some questions with Spark Listener

RangaReddy — Mon, 05 Feb 2024 04:05:01 GMT

Hi @Meepoljd

Please let me know still you need any help on this issue. If any of the above solutions is helped then mark Accept as Solution.