HDP2.5 DATANODE NODEMANGER on different type hosts

[#1]

We have two types hosts.

Type 1: 64 G memory , 11*900G disks

Type 2: 1T memory, 2*500G disks

The interconnect network is 10GE.

So is following deploy architecture reasonable? Is there any potential risk? Any recommendations? Thanks

DATANODE only on Type 1 hosts

RegionServer also only on Type 1 hosts

NODEMANGER only on Type 2 hosts.

others masters are also on Type 2 hosts.

[#2 · Accepted solution]

@Huahua Wei

typically you'd deploy worker (nodemanager) and storage (datanode) service on same nodes to take advantage of having compute close to the data.

[#3]

Hi Artem, Thanks for your quick response.

Currently, we consider to add some disks to Type 2 hosts(but it still less that the Type 1 hosts) and then add datanode on Type 2 hosts . Can this make the compute close to data? And the HDFS on both 2 types hosts with different size local file systems, Can this acceptable?

[#4]

that can work. What I mean compute close to data means both nodemanager and datanode should be on same node.