Am not feeling good to say this. But am not satisfied with you answer. It is fine that application master doing the job of calling inputformat() adn calcuating the input splits and goes on. But am asking what is the meaning of the sentence quoted in the Definitive guide that client places computed inputsplits in HDFS. Am sorry if i am unable to explain my doubt properly. ... View more

@Shiv kumar That is what am saying. So " where this happens? " is my question. ... View more

Very neatly explained.! ... View more

so is it like it will read all the data 1GB and then split the data into logical splits and assign map task to it?? Then what are the computed input splits placed in HDFS while job being submitted... at that AppMaster will not be even launched. and how come 1 GB file will be divided into 10 splits if the block size is 256?? the division is based on splitsize which can be configurable (as of my knowledge). ... View more

YARN has many advantages over MapReduce (MRv1). 1) Scalability - Decreasing the load on the Resource Manager(RM) by delegating the work of handling the tasks running on slaves to application Master, RM can now handle more requests than Job tracker facilitating addition of more nodes. 2) Unlike MPv1 which is strongly coupled with the MapReduce , YARN supports many kinds of code running on them like MR2,Tez, Storm, Spark etc 3) Optimized resource allocation - There are no fixed number of slots separately allocated for Mapper and Reducers in YARN, which is the case in MRv1. So the available capacity of the nodes can be used to any task which needs resources. 4) When Resource manager fails , the jobs running on the cluster need not be restarted again after the recovery of Resource Manager. 5) Failover mechanism is implemented by ZK which is already part of Resource manager which says, we don't need to run another deamon. ... View more

@Benjamin Leonhard Thank you for your quick and explanatory answers. Can you please clarify few more doubts i have, 1) What is the reason behind storing the output of MapReduce to HDFS?? why cant we directly send to client or display them. What happens to the output files?? are they stored permanently or flushed after some time?? if so on what basis?? 2) Will MapReduce run when we read the data from the HDFS?? ... View more

@Benjamin Leonhardi satisfied with your answer. But for the second question , am taking about each chunk of file divided. not about replicas of the block. ... View more