Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

avatar
Expert Contributor

Hi,

I am unable to launch more spark jobs on my cluster due to the error message below. I still have 2.22TB free according to YARN UI. I run HDP 2.6.

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.

What's the way forward?

@Jay Kumar SenSharma

4 REPLIES 4

avatar
Master Mentor

@Joshua Adeleke

The message which you shared normally indicates that your process is not having enough memory. (It is not about the Disk Space)

So can you please share the exact JOB which you are running and where exactly do you see the above kind of message? Like do you see any "hs_err_pid*" file created on the problematic host?

Where exactly do you see this message and can you please share the complete message ?

Also please let us know about the size of your spark executor memory. Can you please try reducing the spark executor memory and then try again because it looks like we see Unable to create new threads means we might need to reduce the Heap a bit.

avatar
Expert Contributor

@Jay Kumar SenSharma

Yes, i see "hs_err_pid*" file. Please find attached the log. Also, the 2.22TB is the un-utilised RAM.

hs-err-pid931.txt

.

avatar
Explorer

@Joshua Adeleke: I'm asking this out of curiosity, how does reducing heap space would provide enough memory to create a new thread?

avatar
Expert Contributor

@rabbit s Reducing the memory specs for the spark executors will reduce the total memory consumed which should eventually allow for more jobs (new threads) to be spun...