Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

Highlighted

There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

Expert Contributor

Hi,

I am unable to launch more spark jobs on my cluster due to the error message below. I still have 2.22TB free according to YARN UI. I run HDP 2.6.

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.

What's the way forward?

@Jay Kumar SenSharma

4 REPLIES 4
Highlighted

Re: There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

Super Mentor

@Joshua Adeleke

The message which you shared normally indicates that your process is not having enough memory. (It is not about the Disk Space)

So can you please share the exact JOB which you are running and where exactly do you see the above kind of message? Like do you see any "hs_err_pid*" file created on the problematic host?

Where exactly do you see this message and can you please share the complete message ?

Also please let us know about the size of your spark executor memory. Can you please try reducing the spark executor memory and then try again because it looks like we see Unable to create new threads means we might need to reduce the Heap a bit.

Highlighted

Re: There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

Expert Contributor

@Jay Kumar SenSharma

Yes, i see "hs_err_pid*" file. Please find attached the log. Also, the 2.22TB is the un-utilised RAM.

hs-err-pid931.txt

.

Highlighted

Re: There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

Explorer

@Joshua Adeleke: I'm asking this out of curiosity, how does reducing heap space would provide enough memory to create a new thread?

Highlighted

Re: There is insufficient memory for the Java Runtime Environment to continue. Cannot create GC thread. Out of system resources.

Expert Contributor

@rabbit s Reducing the memory specs for the spark executors will reduce the total memory consumed which should eventually allow for more jobs (new threads) to be spun...

Don't have an account?
Coming from Hortonworks? Activate your account here