Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

New Yarn node makes the spark driver fail

New Yarn node makes the spark driver fail

Explorer

Hello,

after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:

 

Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster

 

 

This happened only when the driver was started on one of the new nodes.

 

We are submitting with --master yarn-cluster

 

Thanks!

1 REPLY 1
Highlighted

Re: New Yarn node makes the spark driver fail

Master Collaborator
It sounds like that node has not actually been set up as a YARN node
correctly. Make sure it's configured as a NodeManager correctly in CM,
and reports that its health is good and config is up to date.