Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

How to make Yarn deploy resources to new added node?

avatar
Contributor

I added some new nodes to my cluster and it works fine. Then I add Spark Gateway roles to all the new nodes. We're using Yarn to manage and distribute Spark work.

Does adding Spark Gateway roles to new nodes enough to make Yarn think like "Hey there are some new nodes here, let's distribute some containers and work to these new nodes"? Or do I have to add Yarn Gateway roles to these new nodes too?

How to make sure that Yarn will use these new nodes when executing jobs to reduce the overall workload of my cluster

quangbilly79_0-1690257406237.png

 

 

1 ACCEPTED SOLUTION

avatar
Master Collaborator

The node must have a NodeManager role to take part of the processing, Spark gateway, and Yarn Gateway

View solution in original post

2 REPLIES 2

avatar
Master Collaborator

When we submit the spark using YARN, based on YARN resources application will run. In your case you need to add more YARN Gateway nodes to process with more resources. We can't process the data by only added new nodes and yarn will distribute processing all nodes.

avatar
Master Collaborator

The node must have a NodeManager role to take part of the processing, Spark gateway, and Yarn Gateway