Support Questions

Find answers, ask questions, and share your expertise

What are the best config memory and vcore in YARN and mapred multi-node cluster?

avatar
New Contributor

I have a Hadoop cluster with these specifications:

 

PC1 --> master: Number of cores: 40 RAM 64G SSD250G
PC2 --> slave: Number of cores: 30 RAM 32G SSD128G
PC3 --> slave2: Number of cores: 30 RAM 32G SSD128G

 

in each slave, I'll create 2 containers for data nodes, at a total of 5 nodes will setup. Please guide me on what should be the best configuration for memory and vcore on yarn-site.xml and mapred-site.xml for better performance.

thanks

 

1 REPLY 1

avatar
Expert Contributor

As a rule of thumb assign 80% of the node resources to YARN.
Please go through this to modify the config according to your need.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Prop...