question Re: spark.yarn.executor.memoryOverhead in Support Questions

spark.yarn.executor.memoryOverhead

joyabrata — Fri, 16 Sep 2022 12:14:38 GMT

Got below error

17/09/12 20:41:36 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
17/09/12 20:41:39 ERROR cluster.YarnClusterScheduler: Lost executor 1 on xyz.com: remote Akka client disassociated

Please help as not able to find spark.executor.memory or spark.yarn.executor.memoryOverhead in Cloudera Manager (Cloudera Enterprise 5.4.7)

Re: spark.yarn.executor.memoryOverhead

mbigelow — Thu, 14 Sep 2017 18:22:12 GMT

These can be set globally, try searching for just spark memory as CM doesn't always include the actual setting name.

These can be set per job as well. Spark-submit --executor-memory

https://spark.apache.org/docs/1.6.0/submitting-applications.html

Re: spark.yarn.executor.memoryOverhead

ebeb — Fri, 15 Sep 2017 01:43:57 GMT

spark.executor.memory can be found in Cloudera Manager under Hive->configuration and search for Java Heap.

Spark Executor Maximum Java Heap Size
spark.executor.memory
HiveServer2 Default Group

256 MiB

Spark Driver Maximum Java Heap Size
spark.driver.memory
HiveServer2 Default Group

256 MiB

Re: spark.yarn.executor.memoryOverhead

joyabrata — Fri, 15 Sep 2017 11:37:05 GMT

Thank you.
Additional query, do you know why these spark configs are placed under hive?

Re: spark.yarn.executor.memoryOverhead

Yuexin Zhang — Sun, 17 Sep 2017 05:27:35 GMT

It's a spark side configuraion. So you can always specify it via "--conf" option with spark-submit, or you can set the property globally on CM via "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", so CM will include such setting for you via spark gateway client configuration.

Re: spark.yarn.executor.memoryOverhead

AKR — Sun, 05 Jan 2020 14:33:57 GMT

Hi,

This parameter spark.executor.memory (or) spark.yarn.executor.memoryOverhead can be set in Spark submit command or you can set it Advanced configurations.

Thanks

AKR