question Re: Hadoop + Spark Use Case in Support Questions

Hadoop + Spark Use Case

prodgers125 — Thu, 26 May 2016 02:12:35 GMT

There exists some use case that shows how Hadoop and Spark work together? I already read the theory but I want to see something pratical to have a better understand. Thanks!!!

Re: Hadoop + Spark Use Case

bhopp — Thu, 26 May 2016 03:24:06 GMT

There is a good blog post over at MapR regarding this. I personally think the Network Security use case is especially compelling.

https://www.mapr.com/blog/game-changing-real-time-use-cases-apache-spark-on-hadoop.

Re: Hadoop + Spark Use Case

ravi1 — Thu, 26 May 2016 03:36:43 GMT

If you think Hadoop to be HDFS and YARN, spark can take advantage of HDFS (storage that can be horizontally expanded by adding more nodes) by reading data that is in HDFS, writing final processed data into HDFS and YARN (compute that can be horizontally expanded by adding more nodes) by running on YARN.

If you are looking at usecases, look at MLlib algorithms which cover a lot of use cases that can run on top of spark.

Re: Hadoop + Spark Use Case

TimothySpann — Thu, 26 May 2016 03:55:16 GMT

Spark and Hadoop go together like peanut butter and jelly.

Check out my slides

https://community.hortonworks.com/content/idea/28342/apache-zeppelin-with-scala-spark-introduction-to-r.html

https://community.hortonworks.com/content/kbentry/34784/data-ingest-with-apache-zeppelin-apache-spark-16-h.html

I worked at a few places that used Spark and Spark streaming to ingest data into HDFS and HBase. Then Spark + Spark MLib and H20 to run machine learning on the data. Then Hive and Spark SQL for queries. And reporting through Hive Thrift server to Tableau.

Spark without Hadoop really is missing out a lot.

And Spark 1.6 on HDP you get all the benefits of running YARN applications, common security and locality of data access.

I wouldn't run Spark without Hadoop unless you are running Spark standalone for development.

Even there Zeppelin + Spark 1.6 on HDP is an awesome development environment.

Re: Hadoop + Spark Use Case

prodgers125 — Sun, 29 May 2016 06:28:58 GMT

Yes when I think about Hadoop I'm saying to storage the data into HDFS. I don't know what type of advantage that can I take with Spark. Data cleansing?