question Spark on YARN vs Mesos? in Archives of Support Questions (Read Only)

Spark on YARN vs Mesos?

abajwa — Tue, 08 Dec 2015 13:29:29 GMT

What are the considerations for running Spark on YARN vs Spark on Mesos?

Re: Spark on YARN vs Mesos?

nsabharwal — Tue, 08 Dec 2015 20:23:18 GMT

@Ali Bajwa

Spark on YARN

4X usage compared to on MESOS*

* http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/Spark-Survey-2015-Infographic.pdf?t=1443116345933

Kerberos Support

Dynamic Executor Allocation – Scale up & down

Dynamic Executor Allocation with Data locality (Hortonworks Eng added Data locality to dynamic executor allocation)

Better cluster utilization - More than one Executor per node

Spark on Mesos

No Kerberos Support

No data locality for Dynamic Executor Allocation

Dynamic executor Allocation – Only scale down***

Inefficient cluster utilization – limits one executor per slave****

http://spark.apache.org/docs/latest/running-on-mes...

http://apache-spark-developers-list.1001551.n3.nab...

Re: Spark on YARN vs Mesos?

stevel — Thu, 10 Dec 2015 03:17:20 GMT

A key one is straightforward: HDFS is where the data is. YARN schedules work by that data. YARN clusters are very widely deployed, Spark on YARN lets you run Spark queries against that cluster without you even needing to ask permissions from the cluster opts team. To them, it's just another client job.

Re: Spark on YARN vs Mesos?

TimothySpann — Tue, 18 Oct 2016 22:34:03 GMT

any updates for late 2016 on this? Spark 2 support? 1.6.2?

Re: Spark on YARN vs Mesos?

TimothySpann — Tue, 18 Oct 2016 22:34:03 GMT

any updates for late 2016 on this? Spark 2 support? 1.6.2?