Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark\yarn disable second attempt on failure

avatar
Rising Star

when submitting a job that failed, it try to run with another attemped, how i can disable the second run ?


what configuration param i must set?

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Ilia K

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.


View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Ilia K

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.


avatar
Rising Star

i want to disable second attempt

avatar
Master Mentor

@Ilia K

While submitting your Spark Job csan you try passing the

# spark-submit  --master yarn --deploy-mode cluster   --conf spark.yarn.maxAppAttempts=1. .............

.

Or try setting "yarn.resourcemanager.am.max-attempts" to 1 (default may be 2) in Ambari UI --> yarn --> Configs --> Advanced --> Advanced yarn-site

avatar
Rising Star

i think that did what i needed. thanks