Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

Spark app throwing java.lang.OutOfMemoryError: GC overhead limit exceeded

avatar
author-rank buntu
Expert Contributor

Created on ‎08-04-2014 10:38 AM - edited ‎09-16-2022 02:04 AM

I got a 40 node cdh 5.1 cluster and attempting to run a simple spark app that processes about 10-15GB raw data but I keep running into this error:

 

  java.lang.OutOfMemoryError: GC overhead limit exceeded

 

Each node has 8 cores and 2GB memory. I notice the heap size on the executors is set to 512MB with total set to 2GB. Wanted to know whats the heap size needs to be set to for such data sizes?

 

Thanks for the input!

31,264
0 Kudos
3
Who agreed with this topic