What are the best config memory and vcore in YARN and mapred multi-node cluster?

JSDeveloper — Wed, 27 Jul 2022 09:26:40 GMT

I have a Hadoop cluster with these specifications:

PC1 --> master: Number of cores: 40 RAM 64G SSD250G
PC2 --> slave: Number of cores: 30 RAM 32G SSD128G
PC3 --> slave2: Number of cores: 30 RAM 32G SSD128G

in each slave, I'll create 2 containers for data nodes, at a total of 5 nodes will setup. Please guide me on what should be the best configuration for memory and vcore on yarn-site.xml and mapred-site.xml for better performance.

thanks

Re: What are the best config memory and vcore in YARN and mapred multi-node cluster?

AyazHussain — Mon, 22 Apr 2024 05:02:31 GMT

As a rule of thumb assign 80% of the node resources to YARN.
Please go through this to modify the config according to your need.

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Properties

question Re: What are the best config memory and vcore in YARN and mapred multi-node cluster? in Support Questions

What are the best config memory and vcore in YARN and mapred multi-node cluster?

Re: What are the best config memory and vcore in YARN and mapred multi-node cluster?