Our spark jobs are failing with "Received launch task command but executor was null" error. As a work around, we increased the spark.yarn.maxexecutorfailures from 3 to 300 and the job went successful after several executor failures.
Could any one let us know the reasons behind this failure.
We are using Spark 1.6.3 in HDP2.4. We are observing these failures since we upgraded the RAM of our hadoop cluster servers.