Documentation says, map tasks are run in DataNodes and have data locality constraints which the scheduler tries to honor and reduce tasks can run anywhere in the cluster. The statement "can run anywhere in the cluster" for reduce tasks, is referring to only DataNodes in the cluster OR is ResourceManager machine also considered part of the cluster, so that Reduce tasks are allowed to run in ResourceManager also ?? ... View more

Each map task has circular buffer that it writes the output to. The buffer is 100MB by default( the size can be tuned by changing the mapreduce.task.io.sort.mb property). When the contents of the buffer reach a certain threshold size, controlled by a property namely mapreduce.map.sort.spill.percent, with a default value of 80%, a background thread will start to spill the contents to disk. Spills are written in round robin fashion to the directories specified by the mapreduce.cluster.local.dir property in a job specific directory. Where is the Spill directory mentioned in the above property mapreduce.cluster.local.dir located ? 1) Local to ResourceManager machine 2) Local to DataNode machine on which the particular map program runs 3) Local to any DataNode machine otherthan the DataNode machine on which the particular map program runs ... View more

In Hadoop1, horizontal scaling of NameNode was not possible. Due to this there was a limit to which one could perform Horizontal scaling for DataNodes in the cluster. Hadoop2 allows horizontal scaling of NameNodes using a technique called HDFS federation. Without HDFS federation, one cannot add more data nodes to the hadoop cluster beyond a certain limit. In other words, a single namenode's RAM will become fully occupied with metadata/namespace information used for managing the entire datanodes in a cluster. It is at this point that addition of new datanodes, i.e horizontal scaling of data nodes became impossible. Because, namenodes's RAM is already full and cannot handle the new set of metadata/namespace informations, generated as a result of managing the additional data nodes in the cluster. So the limiting factor is the memory capacity of a NameNodes RAM. HDFS federation comes to its rescue by dividing the metadata/namespace across multiple Namenodes, thereby offloading part of one application's metadata/namespace information that have grown beyond the capacity of a single NameNode's RAM to a second NameNode's RAM. Is my understanding correct ?? If YES, then as per documentation, it says that NameNodes are independent and does not communicate each other. In that case who is managing an application's metadata/namespace informations that are resident between two NameNodes? Some sort of daemon should be there that communicates with all NameNodes and finding the portions of metadata split between NameNodes. If NO, then one application's metadata/namespace information does not get split, rather it stays in the same NameNode's RAM only. The next namenode will be used only when the system visualises a situation where an applications's metadata/namespace informations cannot be fully accomodated in its RAM for successfully executing all of its MapReduce jobs. If it does not fit, then system has to try a NameNode whose RAM capacity is sufficient enough to load the job. So my take here is HDFS federation is just a work around and has not actually solved the issue. Please confirm. ... View more