Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (2)
avatar

PROBLEM:

I have just started with livy through Zeppelin and have made some livy interpreter config changes i.e. spark memory. When running

%livy.spark
sc.version

I am getting

Cannot start spark

SOLUTION:

When checking the application log from RM UI I noticed the following

...
End of LogType:launch_container.sh

LogType:stderr
Log Upload Time:Tue Apr 18 15:13:02 +0200 2017
LogLength:142
Log Contents:
Invalid maximum heap size: -Xmx0m
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
...

As you can see, the Xmx value is set to 0m. Doing the further research I noticed the following in livy interpreter:

livy.spark.executor.memory 512
livy.spark.driver.memory 512

Seting the values to

livy.spark.executor.memory 512m 
livy.spark.driver.memory 512m 

saving the changes and restarting livy interpreter fixed the problem

1,071 Views
0 Kudos