Some questions with Spark Listener

[#1]

Hi,recently I have been trying to track my spark application using SparkListener.

Now I have a problem:

         There is a application, the result of its execution is SUCCESSED on Yarn WebUI,but it's actually a failed application.

         And my listener class can not get the error message,the application log only show the Start events for failed jobs and tasks.

Some can help me ?

[#2 · Accepted solution]

Hi @Meepoljd 

Have you checked any exceptions/warnings in the spark application logs and event logs. And also check you listeners are implemented properly and spark application handled exceptions correctly. 

[#3]

On 

onStageCompleted

you should have also the failures and the info on why it failed with StageInfo.failureReason()

Check on the standard event log JSON if the logger stored that event, if it also does not have it, then it means your listener might have an issue. 

If you are not really seeing the StageCompleted event anywhere ( not even on the event log), it probably means the failure was more than just a regular error and it had a major error/crash that aborted the execution of that stage in a way that no StageCompleted was ever sent.

Also bear in mind that there are specific listeners for SQL execution with fail/success events: QueryExecutionListener .

[#4]

Hi @Meepoljd 

Please let me know still you need any help on this issue. If any of the above solutions is helped then mark Accept as Solution.