Segregation between compute node and data node

[#1]

Hi team,

Could you please tell me why do we segregate the compute node from storage node in the hadoop world, In this way are we not breaking the data locality's  philosophy so in this way we are achieving the intra/inter rack data locality not the local data locality and what is the hurdle we faced in the previous design putting the both(compute and data node) on the same node).

Thanks

[#2]

@Faizan123 We are not segregating compute node and data node. Compute node is a node manager and data node is used for storage. If you submit the job the yarn will try to create the task containers on the node where the data is located. The name we use node manager or compute node is used by yarn containers for processing the data. The name data node is used for storing the data. Both can be in a single node.

Please let me know if you have any queries. Also mark "Accept as Solution" if my answer helps you!

Thanks

Shobika S

[#3]

Hi,

But I have seen the data node does not have the node manager roles to process that data that it has, and the node manager nodes does not have the data node role on it,yes but we may have two roles on the same node but in the current time these two roles are running on two different nodes ,so processing through the node manager is happening on the node manager only .

[#4]

@Faizan123, has any of the replies helped resolve your issue? If so, can you kindly mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future? 

[#5]

Hi @Faizan123, I hope the replies provided by @Shelton or @shobikas  has helped you resolve your issue. If so, can you kindly accept them as a solution?