Hadoop nodes with different characteristics

[#1]

Good afternoon, I wanted to know how they could have a cluster with nodes of different characteristics in RAM, for example: slave1: 12 GB RAM

slave2: 12 GB RAM

Slave3: 32 GB RAM

Slave4: 32 GB RAM

The nodes would slave1 and slave2 a group and slave3 and slave4 nodes are another group. The problem is the time to set the parameter of Yarn "yarn.nodemanager.resource.memory-mb" and I do not know if I set it to the first group or the second group .. ??

[#2 · Accepted solution]

you need to leverage ambari config groups @Angel Chiluisa you have to create configuration groups for each hardware profile and then you can assign group specific properties. It is available in the ambari configuration page, look for configuration groups option. Here is a possible link for info. Here is official doc for this.

[#3]

you can toggle which group you assigned what to on the top there's default group then you can toggle the other group. You can also use the filter for the property and find all available configurations. Filter text box is in the right hand corner. @Angel Chiluisa it will show you both group configurations

[#4]

Excellent, thank you very much, I was looking for this configuration

[#5]

glad to help @Angel Chiluisa!

[#6]

I have a similar question, if different workers have different storage capacities or even CPU speeds, what is the impact on the overall cluster? We have been advised to keep all our worker nodes the same spec, but we are not really sure why. How are we going to cope when we have a hardware refresh?

[#7]

The primary design choice to make here is whether we need CPU scheduling (DRF) or not.

In some clusters with varying CPU capacities, due to throughput differences we may need to tweak time out settings to increase them. This is because Network socket time out occur in heterogeneous clusters.

Another aspect is to ensure that each node has enough memory head room left after Yarn allocation to prevent CPU hangs on less capable nodes. (Typically 80% for Yarn and 20% OS etc) . Since one of the nodes has only 12 GB of RAM, you may also want to closely monitor memory usage of processes especially Ambari agent and Ambari metrics memory usage and monitor if it is growing in size, that Yarn is not aware of.

[#8]

Hi,

we have the same in scenario for a CDH 5.10 cluster, with worker node with different cpu and memory.

Is there a similar solution for CDH distribution?

Thanks in advance.

[#9]

@Ivoz As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

[#10]

@DianaTorresThank you!