Skip to main content
ilia987
ilia987
Silver ⭐️⭐️⭐️
February 20, 2019
Solved

spark\yarn disable second attempt on failure

when submitting a job that failed, it try to run with another attemped, how i can disable the second run ?


what configuration param i must set?

jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
February 20, 2019

@Ilia K

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.


View in thread

4 replies

jsensharma
Platinum ⭐️⭐️
jsensharmaAnswer
Platinum ⭐️⭐️
February 20, 2019

@Ilia K

Do you want to control the number of attempts? If yes then you might be interested in the following property "spark.yarn.maxAppAttempts"

Example:

--conf spark.yarn.maxAppAttempts=1

https://spark.apache.org/docs/latest/running-on-yarn.html

spark.yarn.maxAppAttemptsyarn.resourcemanager.am.max-attempts in YARNThe maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.


Translate
    ilia987
    ilia987Author
    Silver ⭐️⭐️⭐️
    February 20, 2019

    i want to disable second attempt

    Translate
    jsensharma
    Platinum ⭐️⭐️
    jsensharma
    Platinum ⭐️⭐️
    February 20, 2019

    @Ilia K

    While submitting your Spark Job csan you try passing the

    # spark-submit  --master yarn --deploy-mode cluster --conf spark.yarn.maxAppAttempts=1. .............

    .

    Or try setting "yarn.resourcemanager.am.max-attempts" to 1 (default may be 2) in Ambari UI --> yarn --> Configs --> Advanced --> Advanced yarn-site

    Translate
    Terms & ConditionsAccessibility statement