question How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin? in Support Questions

How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin?

mph — Wed, 27 Jul 2016 23:09:53 GMT

Hi All,

I notice that each time I run a Spark script through Zeppelin it utilises the full amount of YARN memory available in my cluster. Is there a way that I can limit / manage memory consumption? Irrelevant of the type of job demands it seems to always use 100% of the cluster.

Thanks,M

Re: How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin?

abajwa — Sun, 18 Aug 2019 12:44:26 GMT

In general you can change the executor memory in Zeppelin by modifying zeppelin-env.sh and including

 export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=1g"

If you are installing Zeppelin via Ambari, you can set this via zeppelin.executor.mem (see screenshot)

You can follow the tutorial here which goes through create queue and configure zeppelin to use it

https://github.com/hortonworks-gallery/ambari-zeppelin-service/blob/master/README.md

Re: How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin?

clukasik — Thu, 28 Jul 2016 01:47:49 GMT

Here's one way: edit ZEPPELIN_JAVA_OPTS in zeppelin-env.sh. You can also leverage dynamic allocation (read this) if you want dynamic scaling.

[root@craig-a-1 conf]# grep "ZEPPELIN_JAVA_OPTS" /usr/hdp/current/zeppelin-server/lib/conf/zeppelin-env.sh
# Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"
export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.4.2.0-258 -Dspark.executor.memory=512m -Dspark.executor.instances=2 -Dspark.yarn.queue=default"
# zeppelin interpreter process jvm options. Default = ZEPPELIN_JAVA_OPTS

Re: How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin?

LesterMartin — Mon, 13 Feb 2017 07:39:05 GMT

I tried the following two suggestions independently (full restarts of Zeppelin each time) with no luck.

First, via Ambari, I changed zeppelin.executor.mem from 512m to 256m and zeppelin.executor.instances from 2 to 1.

Then, again via Ambari, I updated the following snippet of the zeppelin_env_content

from

export ZEPPELIN_JAVA_OPTS="-Dhdp.version={{full_stack_version}} -Dspark.executor.memory={{executor_mem}} -Dspark.executor.instances={{executor_instances}} -Dspark.yarn.queue={{spark_queue}}"

export ZEPPELIN_JAVA_OPTS="-Dhdp.version={{full_stack_version}} -Dspark.executor.memory=512m -Dspark.executor.instances=2 -Dspark.yarn.queue={{spark_queue}}"

Both times, my little 2.5 Sandbox was still running full throttle once I run some code in Zeppelin. If anyone notices what I missed on this, please advise.

Re: How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin?

vjiang — Mon, 13 Feb 2017 10:17:04 GMT

You can probably specify a dedicated queue for Zeppelin/Spark in capacity scheduler configuration and give it a limit, for example 50% of the total resources.

Re: How can I limit the amount of YARN memory allocated to the Spark interpreter in Zeppelin?

LesterMartin — Mon, 13 Feb 2017 22:36:20 GMT

Good point. For my Sandbox testing, I decided to just use the steps provided in http://stackoverflow.com/questions/40550011/zeppelin-how-to-restart-sparkcontext-in-zeppelin to stop the SparkContext when I need to do something outside of Zeppelin. Not ideal, but working good enough for some multi-framework prototyping I'm doing.