Member since
02-10-2018
7
Posts
0
Kudos Received
0
Solutions
07-18-2018
11:40 AM
Have you found the solution??? I am looking for solution for my use-case, please follow the link, https://community.hortonworks.com/questions/203537/container-allocation-by-application-master-in-hado.html https://community.hortonworks.com/questions/174245/container-allocation-container-size-in-hadoop-clus.html
... View more
07-18-2018
11:13 AM
Resource Manager allocates Application-Master for each Application/job. Application Master is responsible for lifetime of your Application/Job. Application-Master negotiates with Resource manager and allocates containers on nodemanagers. I am looking for How can i allocate container on specific Datanode??? Please follow following link for Details, https://community.hortonworks.com/questions/203537/container-allocation-by-application-master-in-hado.html If you have found the solution, Please Share.
... View more
07-18-2018
10:55 AM
As we know Default Minimum Container Size is 1024 MBs Consider Hadoop Cluster <1-Master & 3-Datanodes> Namenode : master (3GB RAM, 2 Cores) Datanodes : data1 (512MB RAM, 1 Core), data2(512MB RAM, 1 Core), data3(1GB RAM, 1 Core). Input File Size : "File"=500MB with Replication factor 3 (All the nodes having all the blocks) Default Block Size = 128 MB therefor "File" will be split into 4-Blocks Due to Repli.Factor = 3, all the nodes will be having all the 4-Blocks i.e 4-Containers. Default Minimum Container Size is 1024 MBs Question1 : On which datanode AM will allocate Container? Question2 : Data1 & Data2 has 512MB of RAM, Does that mean AM will never allocate container on these datanodes???
... View more
Labels:
07-18-2018
10:43 AM
I think you have not understood the Scenario here, I want to allocate Container of MR job on particular Datanode. Can we do that by modifying some parameters.
... View more
02-24-2018
09:00 AM
As we know Default Minimum Container Size is 1024MBs Consider this Scenario : I have setup a Hadoop Cluster <1-Master & 3-Datanodes> in VIRTUAL BOX. Namenode : master (3GB RAM) Datanodes : data1 (512MB RAM), data2(512MB RAM), data3(1GB RAM). Input File Size : 500MB with Replication factor 3 (All the nodes having all the blocks) QUESTION : 1. Now lets say i want to run a Mapreduce Program, Will containers get allocated on data1, data2 ???? According to my results Job created containers on all three datanodes. 2. Is there any way we can specify a Job to run on specific node???????
... View more
Labels:
02-10-2018
08:14 AM
Hello, i want to get details "Block-Locations" of a particular file say abc.csv(500mb)(Cluster: 1NM and 3DNs). when i -put a file it is divided into blocks of default size 64MB and spread across Hadoop Cluster. By using Web interface "http://namenode:50070" we can find out block location across cluster. Also by using command : hadoop fsck <file-pat> -files -blocks -locations But what i am trying to achieve is to get these information through JAVA API or WEB- API. Please let me know the solution if any. Any help will be appretiated.
... View more
Labels: