Member since
07-04-2016
40
Posts
5
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1327 | 09-16-2016 05:31 AM |
08-03-2016
12:37 PM
As per Tez sessions, DAGs submitted within a session are handled by the same AppMaster. Unable to understand how the new application (DAG) is mapped to the already running AppMaster?? Who does it and how?? As per YARN the resource manager is responsible for launching appmasters. How this functionality is eclipsed by Tez?? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Tez
-
Apache YARN
08-03-2016
12:33 PM
Am not feeling good to say this. But am not satisfied with you answer. It is fine that application master doing the job of calling inputformat() adn calcuating the input splits and goes on. But am asking what is the meaning of the sentence quoted in the Definitive guide that client places computed inputsplits in HDFS. Am sorry if i am unable to explain my doubt properly.
... View more
08-01-2016
03:30 PM
@Shiv kumar That is what am saying. So " where this happens? " is my question.
... View more
07-26-2016
12:58 PM
Very neatly explained.!
... View more
07-26-2016
11:19 AM
so is it like it will read all the data 1GB and then split the data into logical splits and assign map task to it?? Then what are the computed input splits placed in HDFS while job being submitted... at that AppMaster will not be even launched. and how come 1 GB file will be divided into 10 splits if the block size is 256?? the division is based on splitsize which can be configurable (as of my knowledge).
... View more
07-26-2016
07:16 AM
If it runs in the Appmaster, what exactly are "the computed input splits" that jobclient stores into HDFS while submitting the Job ??
"Copies the resources needed to run the job, including the job JAR file, the
configuration file, and the computed input splits, to the shared filesystem in a directory
named after the job ID (step 3).". Above is the line form Hadoop Definitive guide. And how map works if the split spans over data blocks in two different data nodes??
... View more
Labels:
- Labels:
-
Apache Hadoop
-
HDFS
07-19-2016
01:00 PM
Can MaintenanceMode be the answer?? if yes what happens when a node is kept in maintenance mode.?? How replication works for the data kept in maintenance mode node.?? what happens when i decomission a data node?? and what happens when i delete a datanode???
... View more
Labels:
- Labels:
-
Apache Hadoop
07-11-2016
11:09 AM
YARN has many advantages over MapReduce (MRv1). 1) Scalability - Decreasing the load on the Resource Manager(RM) by delegating the work of handling the tasks running on slaves to application Master, RM can now handle more requests than Job tracker facilitating addition of more nodes. 2) Unlike MPv1 which is strongly coupled with the MapReduce , YARN supports many kinds of code running on them like MR2,Tez, Storm, Spark etc 3) Optimized resource allocation - There are no fixed number of slots separately allocated for Mapper and Reducers in YARN, which is the case in MRv1. So the available capacity of the nodes can be used to any task which needs resources. 4) When Resource manager fails , the jobs running on the cluster need not be restarted again after the recovery of Resource Manager. 5) Failover mechanism is implemented by ZK which is already part of Resource manager which says, we don't need to run another deamon.
... View more
07-05-2016
12:46 PM
@Benjamin Leonhard Thank you for your quick and explanatory answers. Can you please clarify few more doubts i have, 1) What is the reason behind storing the output of MapReduce to HDFS?? why cant we directly send to client or display them. What happens to the output files?? are they stored permanently or flushed after some time?? if so on what basis?? 2) Will MapReduce run when we read the data from the HDFS??
... View more
07-05-2016
11:04 AM
@Benjamin Leonhardi satisfied with your answer. But for the second question , am taking about each chunk of file divided. not about replicas of the block.
... View more
- « Previous
-
- 1
- 2
- Next »