Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

spark-shell java.lang.OutOfMemoryError: Java heap space

Highlighted

spark-shell java.lang.OutOfMemoryError: Java heap space

Explorer

Hello,

 

I work with cloudera-quickstart-vm-5.4.2-0-virtualbox.

 

I built a graph with org.apache.spark.graph

 

Following a message of this community, the command run spark-shell is 

 

[cloudera@quickstart ExamplesOfAnalytics]$ SPARK_REPL_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m"
spark-shell --jars lib/gs-core-1.2.jar,lib/gs-ui-1.2.jar,lib/jcommon-1.0.16.jar,lib/jfreechart-1.0.13.jar,lib/breeze_2.10-0.9.jar,
lib/breeze-viz_2.10-0.9.jar,lib/pherd-1.0.jar

 

Running messages are

16/05/24 11:47:57 INFO executor.Executor: Starting executor ID <driver> on host localhost
16/05/24 11:47:57 INFO executor.Executor: Using REPL class URI: http://10.0.2.15:51787
16/05/24 11:47:57 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@10.0.2.15:57774/user/HeartbeatReceiver
16/05/24 11:47:57 INFO netty.NettyBlockTransferService: Server created on 46454
16/05/24 11:47:57 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/05/24 11:47:57 INFO storage.BlockManagerMasterActor: Registering block manager localhost:46454 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 46454)
16/05/24 11:47:57 INFO storage.BlockManagerMaster: Registered BlockManager
16/05/24 11:47:58 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.

 

I execute the command 

 

 mortagesGraph.vertices.take(5).

 

Error messsage is

 

scala> mortagesGraph.vertices.take(5)
16/05/24 10:08:28 ERROR executor.Executor: Exception in task 0.0 in stage 56.0 (TID 64)
java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newArray(Native Method)
at java.lang.reflect.Array.newInstance(Array.java:70)
at scala.reflect.ClassTag$class.newArray(ClassTag.scala:62)
at scala.reflect.ClassTag$$anon$1.newArray(ClassTag.scala:144)
at org.apache.spark.util.collection.PrimitiveVector.resize(PrimitiveVector.scala:74)
at org.apache.spark.util.collection.PrimitiveVector.$plus$eq(PrimitiveVector.scala:41)
at org.apache.spark.graphx.impl.EdgePartitionBuilder.add(EdgePartitionBuilder.scala:34)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1$$anonfun$apply$1.apply(EdgeRDD.scala:107)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1$$anonfun$apply$1.apply(EdgeRDD.scala:106)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1.apply(EdgeRDD.scala:106)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1.apply(EdgeRDD.scala:104)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

 

 

How can I fix the error?

 

Thanks in advance

 

Carlota Vina