Sorry, this is a bug described in SPARK-22876 which suggests that the current logic of spark.yarn.am.attemptFailuresValidityInterval is flawed. While the jira is still being worked on, looking at the comments, I don't foresee a fix anytime soon.
... View more
