Created 01-02-2021 03:38 PM
Resource manager manages resources by communicating with the node managers .
Are these node managers same as data nodes ?
Also , does the resource manager communicate with node managers via Name Node since it has all the meta data ?
And after resources have been allocated , for a map Reduce Job , the mappers and reducers job is scheduled by Name node , right ?
I got confused when the term resource manager came after Name node , hence looking for a confirmation for the basics
Created 01-02-2021 04:07 PM
Hadoop is designed to ensure that compute (Node Managers) runs as close to data (Data Nodes) as possible. Usually containers for jobs are allocated on the same nodes where the data is present. Hence in a typical Hadoop cluster, both Data Nodes and Node Manager run on the same machine.
Node Manager is the RM slave process while the Data Nodes is the Namenode slave process which responsible for coordinating HDFS functions
Resource Manager: Runs on a master daemon and manages the resource allocation in the cluster. Node Manager: They run on the slave daemons and are responsible for the execution of a task on every single Data Node
Resource manager and Namenode both as master components [processes] that can run in single or HA setup should run on separate identical usually high spec servers [nodes] as compared to the data nodes. Zookeeper is another important component
ResourceManager and NodeManager combine together to form a data-computation framework.
Hope that helps