I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.
this will give you new job id
curl -v -X POST 'http://localhost:8088/ws/v1/cluster/apps/new-application'
get the job id and fire this request to know the status
or if you know the job id already you need to run the the second curl request only.
.Thanks Rajkumar. But,how can i setup alert(email) notification?. I want to know success,failure and logs through email
with mapreduce we have a feature called CallBackMR where application master sends the notification to http webserver when job end by configuring 'job.end.notification.url' but with spark I dont think this feature is available with spark.
to get success,failure and logs you need to write a custom script which run periodically get the job status from rest api and grep the code based on the code it will send email alert.
You can try using oozie. You can create a simple workflow which starts with your Spark application and if it fails send you an email. Look at here for reference: http://oozie.apache.org/docs/4.2.0/index.html