after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:
Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
This happened only when the driver was started on one of the new nodes.
We are submitting with --master yarn-cluster