Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.
after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:
Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
This happened only when the driver was started on one of the new nodes.
It sounds like that node has not actually been set up as a YARN node correctly. Make sure it's configured as a NodeManager correctly in CM, and reports that its health is good and config is up to date.