question spark-shell java.lang.OutOfMemoryError: PermGen space in Archives of Support Questions (Read Only)

spark-shell java.lang.OutOfMemoryError: PermGen space

steff — Fri, 16 Sep 2022 08:57:26 GMT

I'm using the spark-shell to run a linear regression on a small data set with 5000 observations, but I get a "java.lang.OutOfMemoryError: PermGen space" error message. How can I increase the MaxPermSize for the spark-shell?

Thanks.

Stefan

Re: spark-shell java.lang.OutOfMemoryError: PermGen space

srowen — Tue, 15 Apr 2014 06:47:40 GMT

(By the way, there is a separate forum for Spark: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/bd-p/Spark )

If the error is definitely in the shell / REPL, then I believe you just want to set SPARK_REPL_OPTS:

SPARK_REPL_OPTS="-XX:MaxPermSize=256m" spark-shell

I find this other setting can help with permgen usage in Scala, which is an issue when running tests too:

SPARK_REPL_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m" spark-shell

Re: spark-shell java.lang.OutOfMemoryError: PermGen space

steff — Tue, 15 Apr 2014 16:26:10 GMT

Great that solved my problem. i'll use the Spark dedicated forum from now on for Spark related questions thanks for pointing that out.

Thanks for your help.

Stefan