Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

What are the best config memory and vcore in YARN and mapred multi-node cluster?

avatar
New Member

I have a Hadoop cluster with these specifications:

 

PC1 --> master: Number of cores: 40 RAM 64G SSD250G
PC2 --> slave: Number of cores: 30 RAM 32G SSD128G
PC3 --> slave2: Number of cores: 30 RAM 32G SSD128G

 

in each slave, I'll create 2 containers for data nodes, at a total of 5 nodes will setup. Please guide me on what should be the best configuration for memory and vcore on yarn-site.xml and mapred-site.xml for better performance.

thanks

 

1 REPLY 1

avatar
Super Collaborator

As a rule of thumb assign 80% of the node resources to YARN.
Please go through this to modify the config according to your need.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Prop...