Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Some questions with Spark Listener

avatar
Rising Star

Hi,recently I have been trying to track my spark application using SparkListener.

Now I have a problem:

         There is a application, the result of its execution is SUCCESSED on Yarn WebUI,but it's actually a failed application.

         And my listener class can not get the error message,the application log only show the Start events for failed jobs and tasks.

Some can help me ?

Meepoljd_1-1670383825382.png

 

Meepoljd_0-1670383733464.png

 

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Hi @Meepoljd 

Have you checked any exceptions/warnings in the spark application logs and event logs. And also check you listeners are implemented properly and spark application handled exceptions correctly. 

View solution in original post

3 REPLIES 3

avatar
Master Collaborator

Hi @Meepoljd 

Have you checked any exceptions/warnings in the spark application logs and event logs. And also check you listeners are implemented properly and spark application handled exceptions correctly. 

avatar
Contributor

On 

onStageCompleted

you should have also the failures and the info on why it failed with StageInfo.failureReason()

Check on the standard event log JSON if the logger stored that event, if it also does not have it, then it means your listener might have an issue. 

If you are not really seeing the StageCompleted event anywhere ( not even on the event log), it probably means the failure was more than just a regular error and it had a major error/crash that aborted the execution of that stage in a way that no StageCompleted was ever sent.

Also bear in mind that there are specific listeners for SQL execution with fail/success events: QueryExecutionListener .

avatar
Master Collaborator

Hi @Meepoljd 

Please let me know still you need any help on this issue. If any of the above solutions is helped then mark Accept as Solution.