question Re: Can't specify 2G Heap in Archives of Support Questions (Read Only)

Can't specify 2G Heap

wkupersa — Fri, 16 Sep 2022 11:23:44 GMT

I am getting garbage collection errors: "java.lang.OutOfMemoryError: GC overhead limit exceeded" Everything that I have read points to heap size.

I have upped all the heap related parameters that I see in my yarn configuration options.

I am trying to run spark-submit with the argument --driver-java-options " -Xmx2048m" and I get the error

"Initial heap size set to a larger value than the maximum heap size"

I am not sure why it says that the maximum heap size is smaller than 2G? I am not sure what else to look at.

Thanks!

Re: Can't specify 2G Heap

srowen — Sat, 01 Apr 2017 10:16:56 GMT

Don't set heap size this way. Use --driver-memory. This indicates you are actually setting the max heap smaller than your driver memory is configured elsewhere, perhaps in a .conf file.

Re: Can't specify 2G Heap

wkupersa — Mon, 03 Apr 2017 21:23:34 GMT

I tried this and while my job is still running, it looks like it has gotten farther than it has in the past. Thanks!