after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:
Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
This happened only when the driver was started on one of the new nodes.
It sounds like that node has not actually been set up as a YARN node correctly. Make sure it's configured as a NodeManager correctly in CM, and reports that its health is good and config is up to date.