Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎09-27-2017

Spark 2 Release 4- Weird Issue causing java.lang.OutOfMemoryError: GC overhead limit,Java heap space

hello Folks,

 

Running a CDH 5.14.4 cluster along with Spark 2.3 Release 4 CDS. While submitting a fairly large Spark SQL job from a spark application(client mode) , facing some java.lang.OutOfMemoryError errors on few executors. The error occurs is aganostic of the underlying data volume as i am able to see this with 100 or 1 million records. The SQL is quite bulky as it execute a large number of complex conditional statements and expressions.

 

However when i run the application in local mode instead of yarn as the master , execution goes through smoothly.

 

Request immediate assitance from the community here.

 

#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill %p"
# Executing /bin/sh -c "kill 4505"...
15:20:57.430 [SIGTERM handler] ERROR org.apache.spark.executor.CoarseGrainedExecutorBackend - RECEIVED SIGNAL TERM
15:20:57.435 [Thread-2] INFO org.apache.spark.storage.DiskBlockManager - Shutdown hook called
15:20:57.457 [Executor task launch worker for task 13466] ERROR org.apache.spark.executor.Executor - Exception in task 118.0 in stage 146.1 (TID 13466)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:113)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:112)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:112)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:104)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:137)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:138)
at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.logDebug(RuleExecutor.scala:40)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:134)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)


#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill %p"
# Executing /bin/sh -c "kill 8562"...
15:21:11.409 [SIGTERM handler] ERROR org.apache.spark.executor.CoarseGrainedExecutorBackend - RECEIVED SIGNAL TERM
15:21:11.420 [Thread-2] INFO org.apache.spark.storage.DiskBlockManager - Shutdown hook called
15:21:11.454 [Executor task launch worker for task 13470] ERROR org.apache.spark.executor.Executor - Exception in task 129.0 in stage 146.1 (TID 13470)
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430)
at scala.collection.immutable.StringLike$class.$times(StringLike.scala:71)
at scala.collection.immutable.StringOps.$times(StringOps.scala:29)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:113)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:112)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:112)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:104)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:137)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:138)
at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.logDebug(RuleExecutor.scala:40)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:134)

 

Announcements