spark-shell java.lang.OutOfMemoryError: PermGen space

[#1]

I'm using the spark-shell to run a linear regression on a small data set with 5000 observations, but I get a "java.lang.OutOfMemoryError: PermGen space" error message. How can I increase the MaxPermSize for the spark-shell?

Thanks.

Stefan

[#2 · Accepted solution]

(By the way, there is a separate forum for Spark: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/bd-p/Spark )

If the error is definitely in the shell / REPL, then I believe you just want to set SPARK_REPL_OPTS:

SPARK_REPL_OPTS="-XX:MaxPermSize=256m" spark-shell

I find this other setting can help with permgen usage in Scala, which is an issue when running tests too:

SPARK_REPL_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m" spark-shell

[#3]

Great that solved my problem. i'll use the Spark dedicated forum from now on for Spark related questions thanks for pointing that out.

Thanks for your help.

Stefan