Reply
Highlighted
Explorer
Posts: 9
Registered: ‎08-13-2015

New Yarn node makes the spark driver fail

Hello,

after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:

 

Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster

 

 

This happened only when the driver was started on one of the new nodes.

 

We are submitting with --master yarn-cluster

 

Thanks!

Cloudera Employee
Posts: 366
Registered: ‎07-29-2013

Re: New Yarn node makes the spark driver fail

It sounds like that node has not actually been set up as a YARN node
correctly. Make sure it's configured as a NodeManager correctly in CM,
and reports that its health is good and config is up to date.