Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

New Yarn node makes the spark driver fail

New Yarn node makes the spark driver fail



after some time, we decided to add a few nodes to our Yarn cluster. However, after adding the nodes to cloudera and assigning them the Yarn NodeManager role, the spark applications started to fail with:


Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster



This happened only when the driver was started on one of the new nodes.


We are submitting with --master yarn-cluster




Re: New Yarn node makes the spark driver fail

Master Collaborator
It sounds like that node has not actually been set up as a YARN node
correctly. Make sure it's configured as a NodeManager correctly in CM,
and reports that its health is good and config is up to date.

Don't have an account?
Coming from Hortonworks? Activate your account here