This is also happening on Spark Streaming running on top of Hadoop 2.6. YARN restarts the Spark application automatically, but every now and then, Spark's checkpoint gets corrupted, even though graceful shutdown is enabled. I don't see any major fixes for AMRM token issues after Hadoop 2.6.0, so I'm not sure this bug is not the effect of something else that is not visible in the spark driver log. Has anyone else encountered this or has any ideas what might be causing it? Using Spark 1.5.1 and Hadoop 2.6.0, cloudera distributions.
... View more