question Re: Heterogeneous worker nodes with HDP2.3 in Archives of Support Questions (Read Only)

Heterogeneous worker nodes with HDP2.3

bsaini — Fri, 23 Oct 2015 22:07:28 GMT

What precautions & extra configurations, if any, are needed when adding worker nodes with different capacity to a cluster? My understanding is that YARN will be able to just manage the nodes without anything special.

For e.g - Any issues with adding 3 nodes with following config to an existing POC cluster that has similar nodes 8 cores, 32 Gigs, 3 TB DAS for data-

node1 - 8 cores, 64GB RAM, NO STORAGE 

node2 - 8 cores, 64GB RAM, 2 TB

node3 - 8 cores, 64GB RAM, 2TB

Also, how do you configure YARN to utilize different amount of memory on these heterogeneous boxes?

Re: Heterogeneous worker nodes with HDP2.3

aervits — Sat, 24 Oct 2015 08:36:55 GMT

you need to run the configuration utility against these nodes to account for memory, disk and cpu sizing. If you don't put node1 with no storage in its own group and specify not to look for whatever the hdfs or yarn directory you will run out of disk pretty quick because hdfs will try to write data to nonexisting directories in filesystem. So for node 1, I'd put it in it's own Ambari configuration group.

Re: Heterogeneous worker nodes with HDP2.3

nsabharwal — Mon, 19 Aug 2019 12:56:31 GMT

@bsaini@hortonworks.com

Node 1 - no storage - I guess you wont be using this for data node but it can play role of worker node (Nodemanager if you like and other components)

You will be creating a config group for node1

Please see this screenshot. In my case, node4 is datanode and I want to customize the data directories

Created new config for node4

select the config and change parameters that you want to change.

There is

Doc example