question Re: spark\yarn disable second attempt on failure in Support Questions

spark\yarn disable second attempt on failure

ilia987 — Wed, 20 Feb 2019 20:03:03 GMT

when submitting a job that failed, it try to run with another attemped, how i can disable the second run ?

what configuration param i must set?

Re: spark\yarn disable second attempt on failure

jsensharma — Wed, 20 Feb 2019 20:07:18 GMT

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttempts yarn.resourcemanager.am.max-attempts in YARN The maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.

Re: spark\yarn disable second attempt on failure

ilia987 — Wed, 20 Feb 2019 20:15:00 GMT

i want to disable second attempt

Re: spark\yarn disable second attempt on failure

jsensharma — Wed, 20 Feb 2019 20:18:28 GMT

@Ilia K

While submitting your Spark Job csan you try passing the

# spark-submit  --master yarn --deploy-mode cluster   --conf spark.yarn.maxAppAttempts=1. .............

Or try setting "yarn.resourcemanager.am.max-attempts" to 1 (default may be 2) in Ambari UI --> yarn --> Configs --> Advanced --> Advanced yarn-site

Re: spark\yarn disable second attempt on failure

ilia987 — Wed, 20 Feb 2019 21:26:18 GMT

i think that did what i needed. thanks