Reply
Explorer
Posts: 11
Registered: ‎07-16-2015

What are the recommended configuration resources to SPARK?

[ Edited ]

Hi all

 

I need to know what % of resources assigned to Spark.

Actually i assigned 50% Impala 50% Mapreduce but i need to know the advaisable % to Spark,Mapreduce and Impala.

 

Best regards

Posts: 1,885
Kudos: 422
Solutions: 298
Registered: ‎07-31-2013

Re: What are the recommended configuration resources to SPARK?

This is need-dependent. Are you running Spark over YARN or standalone instead? If the former, then the existing 50% covers Spark applications also.
Highlighted
Cloudera Employee
Posts: 65
Registered: ‎09-11-2015

Re: What are the recommended configuration resources to SPARK?

Advanced Apache Spark- Sameer Farooqui (Databricks)
    https://www.youtube.com/watch?v=7ooZ4S7Ay6Y

 

Advance to mark 2:49:30

 

  • Recommended to use at most 75% of a machines' memory for Spark
  • Minimum Executor heap size should be 8GB 
  • Maximum Executor heap size depends.... maybe 40GB (watch GC)
  • Memory usage is greatly affected by storage level serialization format
Live Big Data Training from Spark Summit 2015 in New York City. "Today I'll cover Spark core in depth and get you prepared to use Spark in your own prototypes. We'll start by learning about the big data ecosystem, then jump into RDDs (Resilient Distributed Datasets). Then we'll talk about ...