question Re: HDP2.5 DATANODE NODEMANGER on different type hosts in Archives of Support Questions (Read Only)

HDP2.5 DATANODE NODEMANGER on different type hosts

diablo2 — Tue, 25 Oct 2016 19:07:15 GMT

We have two types hosts.

Type 1: 64 G memory , 11*900G disks

Type 2: 1T memory, 2*500G disks

The interconnect network is 10GE.

So is following deploy architecture reasonable? Is there any potential risk? Any recommendations? Thanks

DATANODE only on Type 1 hosts

RegionServer also only on Type 1 hosts

NODEMANGER only on Type 2 hosts.

others masters are also on Type 2 hosts.

Re: HDP2.5 DATANODE NODEMANGER on different type hosts

aervits — Wed, 26 Oct 2016 15:21:28 GMT

@Huahua Wei

typically you'd deploy worker (nodemanager) and storage (datanode) service on same nodes to take advantage of having compute close to the data.

Re: HDP2.5 DATANODE NODEMANGER on different type hosts

diablo2 — Thu, 27 Oct 2016 14:34:13 GMT

Hi Artem, Thanks for your quick response.

Currently, we consider to add some disks to Type 2 hosts(but it still less that the Type 1 hosts) and then add datanode on Type 2 hosts . Can this make the compute close to data? And the HDFS on both 2 types hosts with different size local file systems, Can this acceptable?

Re: HDP2.5 DATANODE NODEMANGER on different type hosts

aervits — Thu, 27 Oct 2016 17:15:49 GMT

that can work. What I mean compute close to data means both nodemanager and datanode should be on same node.