Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark.yarn.executor.memoryOverhead

avatar
Explorer

Got below error

17/09/12 20:41:36 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

17/09/12 20:41:39 ERROR cluster.YarnClusterScheduler: Lost executor 1 on xyz.com: remote Akka client disassociated

Please help as not able to find spark.executor.memory or spark.yarn.executor.memoryOverhead in Cloudera Manager (Cloudera Enterprise 5.4.7)

1 ACCEPTED SOLUTION

avatar
Expert Contributor

spark.executor.memory  can be found in Cloudera Manager under Hive->configuration and search for Java Heap.

 

Spark Executor Maximum Java Heap Size
spark.executor.memory
HiveServer2 Default Group

256 MiB

Spark Driver Maximum Java Heap Size
spark.driver.memory
HiveServer2 Default Group

256 MiB

View solution in original post

5 REPLIES 5

avatar
Champion
These can be set globally, try searching for just spark memory as CM doesn't always include the actual setting name.

These can be set per job as well. Spark-submit --executor-memory

https://spark.apache.org/docs/1.6.0/submitting-applications.html

avatar
Expert Contributor

spark.executor.memory  can be found in Cloudera Manager under Hive->configuration and search for Java Heap.

 

Spark Executor Maximum Java Heap Size
spark.executor.memory
HiveServer2 Default Group

256 MiB

Spark Driver Maximum Java Heap Size
spark.driver.memory
HiveServer2 Default Group

256 MiB

avatar
Explorer

Thank you.
Additional query, do you know why these spark configs are placed under hive?

avatar
Expert Contributor

It's a spark side configuraion. So you can always specify it via "--conf" option with spark-submit, or you can set the property globally on CM via "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", so CM will include such setting for you via spark gateway client configuration.

avatar
Cloudera Employee

Hi,

 

This parameter spark.executor.memory (or) spark.yarn.executor.memoryOverhead can be set in Spark submit command or you can set it Advanced configurations.

 

Thanks

AKR