Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

Highlighted

Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

New Contributor

I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

5 REPLIES 5
Highlighted

Re: Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

this will give you new job id

curl -v -X POST 'http://localhost:8088/ws/v1/cluster/apps/new-application'

get the job id and fire this request to know the status

curl 'http://localhost:8088/ws/v1/cluster/apps/<job_id>/state'

or if you know the job id already you need to run the the second curl request only.

Highlighted

Re: Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

New Contributor

.Thanks Rajkumar. But,how can i setup alert(email) notification?. I want to know success,failure and logs through email

Highlighted

Re: Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

with mapreduce we have a feature called CallBackMR where application master sends the notification to http webserver when job end by configuring 'job.end.notification.url' but with spark I dont think this feature is available with spark.

to get success,failure and logs you need to write a custom script which run periodically get the job status from rest api and grep the code based on the code it will send email alert.

Re: Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

Rising Star

You can try using oozie. You can create a simple workflow which starts with your Spark application and if it fails send you an email. Look at here for reference: http://oozie.apache.org/docs/4.2.0/index.html

Highlighted

Re: Hi, I have a query in Spark.How to set up email notification when yarn job failed?. Which REST API i can use?. Thanks.

@raja rajeshwaran

You can get application status here :

http://localhost:4040/api/v1/applications/Spark%20shell/jobs

And create a custom logic which can send the status through some smtp service.

Don't have an account?
Coming from Hortonworks? Activate your account here