Reply
New Contributor
Posts: 4
Registered: ‎01-14-2017
Accepted Solution

Can't specify 2G Heap

I am getting garbage collection errors: "java.lang.OutOfMemoryError: GC overhead limit exceeded" Everything that I have read points to heap size.

 

I have upped all the heap related parameters that I see in my yarn configuration options.  

 

I am trying to run spark-submit with the argument --driver-java-options " -Xmx2048m"  and I get the error

"Initial heap size set to a larger value than the maximum heap size"

 

I am not sure why it says that the maximum heap size is smaller than 2G? I am not sure what else to look at. 

 

Thanks!

 

 

Cloudera Employee
Posts: 411
Registered: ‎08-11-2014

Re: Can't specify 2G Heap

Don't set heap size this way. Use --driver-memory. This indicates you are actually setting the max heap smaller than your driver memory is configured elsewhere, perhaps in a .conf file.

Highlighted
New Contributor
Posts: 4
Registered: ‎01-14-2017

Re: Can't specify 2G Heap

I tried this and while my job is still running, it looks like it has gotten farther than it has in the past. Thanks!

Announcements