https://community.cloudera.com/t5/Archives-of-Support-Questions/Multi-threading-in-Hadoop/m-p/106784#M46479 <P>@<A href="https://community.hortonworks.com/users/2270/jeevakrishnarg.html">Jeeva Jeeva</A></P><P><STRONG>Multithreading Programming Model </STRONG>and <STRONG>MapReduce Programming Model</STRONG> are based on fundamentally different principles and both are meant to solve different kinds of data storage and processing problems. Multithreading is based on <STRONG>Parallelization of Processing</STRONG> where as Hadoop takes power by <STRONG>Parallelization of Data</STRONG>.</P><P>If you assume that the Hadoop ecosystem is only MapReduce and Spark batch, then your understanding is correct . However, the ecosystem includes also real-time streaming tools like Apache Storm which uses multi-threading. However, modern tools handle all these programmatic needs for multi-threading by architecture/design. Their focus is scalability by architecture and design and less by laborious programming efforts.</P><P>References: </P><P><A href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/" target="_blank">http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/</A></P><P><A href="https://www.safaribooksonline.com/blog/2014/01/06/multi-threading-storm" target="_blank">https://www.safaribooksonline.com/blog/2014/01/06/multi-threading-storm</A></P><P>+++</P><P>If it helped, pls vote/accept best answer</P> Thu, 17 Nov 2016 06:55:37 GMT cstanca 2016-11-17T06:55:37Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Multi-threading-in-Hadoop/m-p/106783#M46478 <P>Hi there,</P><P>I have an idea about Multi-threading in general but not sure how it is used in Hadoop.</P><P>Based on my knowledge, Yarn is responsible for managing/controlling Spark/Mapreduce job resources, can't think of Multi-threading here. Not sure whether it can be used anywhere else in Hadoop Eco System.</P><P>I would appreciate if anybody could provide some information on this.</P><P>Many thanks,</P> Wed, 16 Nov 2016 21:01:29 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Multi-threading-in-Hadoop/m-p/106783#M46478 JeevaKrish 2016-11-16T21:01:29Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Multi-threading-in-Hadoop/m-p/106784#M46479 <P>@<A href="https://community.hortonworks.com/users/2270/jeevakrishnarg.html">Jeeva Jeeva</A></P><P><STRONG>Multithreading Programming Model </STRONG>and <STRONG>MapReduce Programming Model</STRONG> are based on fundamentally different principles and both are meant to solve different kinds of data storage and processing problems. Multithreading is based on <STRONG>Parallelization of Processing</STRONG> where as Hadoop takes power by <STRONG>Parallelization of Data</STRONG>.</P><P>If you assume that the Hadoop ecosystem is only MapReduce and Spark batch, then your understanding is correct . However, the ecosystem includes also real-time streaming tools like Apache Storm which uses multi-threading. However, modern tools handle all these programmatic needs for multi-threading by architecture/design. Their focus is scalability by architecture and design and less by laborious programming efforts.</P><P>References: </P><P><A href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/" target="_blank">http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/</A></P><P><A href="https://www.safaribooksonline.com/blog/2014/01/06/multi-threading-storm" target="_blank">https://www.safaribooksonline.com/blog/2014/01/06/multi-threading-storm</A></P><P>+++</P><P>If it helped, pls vote/accept best answer</P> Thu, 17 Nov 2016 06:55:37 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Multi-threading-in-Hadoop/m-p/106784#M46479 cstanca 2016-11-17T06:55:37Z