New Yarn node makes the spark driver fail

[#1]

Hello,

after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:

Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster

This happened only when the driver was started on one of the new nodes.

We are submitting with --master yarn-cluster

Thanks!

[#2]

It sounds like that node has not actually been set up as a YARN node
correctly. Make sure it's configured as a NodeManager correctly in CM,
and reports that its health is good and config is up to date.