question Re: Will the Spark replace the Map-Reduce in Hadoop? in Archives of Support Questions (Read Only)

Will the Spark replace the Map-Reduce in Hadoop?

vinayreddy536 — Sun, 12 Jun 2016 14:57:19 GMT

As Spark has much more advantages over the Map-Reduce framework like in-memory processing, Faster Real-time data analysis and also it claims to process data 100x faster than MapReduce, while 10x faster with the disks. So, Will it be replacing the Map-Reduce? or will it be used paralelly?

Re: Will the Spark replace the Map-Reduce in Hadoop?

rajkumar_singh — Sun, 12 Jun 2016 15:22:04 GMT

@Vinay Reddy NaguruSaprk is far ahead in terms of perfomance but it still need to address some of the concern like memory management. for some use case mapreduce is preferred over spark e.g. ETL calculations where result sets are vast and may exceed total RAM of hadoop cluster,mapreduce can out perform Spark for this situation.Iterative machine learning where spark is not able to manage memory more proficiently is ideal use case for mapreduce. but spark is evolving so fast and trying address these concerns so i think it is not very distant when spark will replace MR completely but for now they can coexists in the cluster.

Re: Will the Spark replace the Map-Reduce in Hadoop?

TimothySpann — Sun, 12 Jun 2016 18:17:10 GMT

More likely that principals and techniques from spark and flink will enhance map reduce. Flink is faster so that would've a better choice than Spark. Tez is very powerful accelerator

Re: Will the Spark replace the Map-Reduce in Hadoop?

vinayreddy536 — Sun, 12 Jun 2016 18:32:31 GMT

Thank you for your response @Timothy Spann