Support Questions
Find answers, ask questions, and share your expertise

spark\yarn disable second attempt on failure

Explorer

when submitting a job that failed, it try to run with another attemped, how i can disable the second run ?


what configuration param i must set?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: spark\yarn disable second attempt on failure

Super Mentor

@Ilia K

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.


View solution in original post

4 REPLIES 4

Re: spark\yarn disable second attempt on failure

Super Mentor

@Ilia K

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.


View solution in original post

Re: spark\yarn disable second attempt on failure

Explorer

i want to disable second attempt

Re: spark\yarn disable second attempt on failure

Super Mentor

@Ilia K

While submitting your Spark Job csan you try passing the

# spark-submit  --master yarn --deploy-mode cluster   --conf spark.yarn.maxAppAttempts=1. .............

.

Or try setting "yarn.resourcemanager.am.max-attempts" to 1 (default may be 2) in Ambari UI --> yarn --> Configs --> Advanced --> Advanced yarn-site

Re: spark\yarn disable second attempt on failure

Explorer

i think that did what i needed. thanks