https://community.cloudera.com/t5/Support-Questions/how-to-set-and-change-the-number-of-parallel-tasks-in-a/m-p/213811#M175735 <P>Hi everybody, i want to do DFSIO test on my hadoop cluster. I wanna know which parameter, show the number of parallel task on a data node? i mean what is the name of the parameter and where is it? i want to change it , and analyze the results. thank u very much.</P><P>i have read an article, in this link;</P><P><A href="https://pdfs.semanticscholar.org/e874/ed1a4b120bb4caf5c2ecb440f49df7819d54.pdf" target="_blank">https://pdfs.semanticscholar.org/e874/ed1a4b120bb4caf5c2ecb440f49df7819d54.pdf</A></P><P>In this article, they use these parameters for DFSIO test:</P><LI> BufferSize: 1000000 bytes</LI><LI>Replication factor: 3 </LI><LI> Number of tasks in parallel by node: 2 </LI><LI> Block size: 64 MB </LI><LI>what is the "Number of tasks in parallel by node" , and where can i set it in hortonworks hadoop?</LI> Fri, 13 Apr 2018 19:42:06 GMT ma_namvar 2018-04-13T19:42:06Z https://community.cloudera.com/t5/Support-Questions/how-to-set-and-change-the-number-of-parallel-tasks-in-a/m-p/213811#M175735 <P>Hi everybody, i want to do DFSIO test on my hadoop cluster. I wanna know which parameter, show the number of parallel task on a data node? i mean what is the name of the parameter and where is it? i want to change it , and analyze the results. thank u very much.</P><P>i have read an article, in this link;</P><P><A href="https://pdfs.semanticscholar.org/e874/ed1a4b120bb4caf5c2ecb440f49df7819d54.pdf" target="_blank">https://pdfs.semanticscholar.org/e874/ed1a4b120bb4caf5c2ecb440f49df7819d54.pdf</A></P><P>In this article, they use these parameters for DFSIO test:</P><LI> BufferSize: 1000000 bytes</LI><LI>Replication factor: 3 </LI><LI> Number of tasks in parallel by node: 2 </LI><LI> Block size: 64 MB </LI><LI>what is the "Number of tasks in parallel by node" , and where can i set it in hortonworks hadoop?</LI> Fri, 13 Apr 2018 19:42:06 GMT https://community.cloudera.com/t5/Support-Questions/how-to-set-and-change-the-number-of-parallel-tasks-in-a/m-p/213811#M175735 ma_namvar 2018-04-13T19:42:06Z https://community.cloudera.com/t5/Support-Questions/how-to-set-and-change-the-number-of-parallel-tasks-in-a/m-p/213812#M175736 <P>Like other MR jobs, you can specify the number of map tasks for the job via JobConf.setNumMapTasks(). However, this is only a hint and the actual number of spawned map tasks depends on the number of input splits. If you have set NumMapTasks=100 with 50 nodes, then the number of tasks in parallel per node is about 100/50 = 2 assuming the splits are evenly distributed across the nodes.</P> Sat, 14 Apr 2018 05:56:25 GMT https://community.cloudera.com/t5/Support-Questions/how-to-set-and-change-the-number-of-parallel-tasks-in-a/m-p/213812#M175736 xyao 2018-04-14T05:56:25Z