Posts: 9
Registered: ‎08-13-2015

New Yarn node makes the spark driver fail


after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:


Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster



This happened only when the driver was started on one of the new nodes.


We are submitting with --master yarn-cluster



Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: New Yarn node makes the spark driver fail

It sounds like that node has not actually been set up as a YARN node
correctly. Make sure it's configured as a NodeManager correctly in CM,
and reports that its health is good and config is up to date.