Reply
Explorer
Posts: 13
Registered: ‎05-25-2016

spark-shell java.lang.OutOfMemoryError: Java heap space

Hello,

 

I work with cloudera-quickstart-vm-5.4.2-0-virtualbox.

 

I built a graph with org.apache.spark.graph

 

Following a message of this community, the command run spark-shell is 

 

[cloudera@quickstart ExamplesOfAnalytics]$ SPARK_REPL_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m"
spark-shell --jars lib/gs-core-1.2.jar,lib/gs-ui-1.2.jar,lib/jcommon-1.0.16.jar,lib/jfreechart-1.0.13.jar,lib/breeze_2.10-0.9.jar,
lib/breeze-viz_2.10-0.9.jar,lib/pherd-1.0.jar

 

Running messages are

16/05/24 11:47:57 INFO executor.Executor: Starting executor ID <driver> on host localhost
16/05/24 11:47:57 INFO executor.Executor: Using REPL class URI: http://10.0.2.15:51787
16/05/24 11:47:57 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@10.0.2.15:57774/user/HeartbeatReceiver
16/05/24 11:47:57 INFO netty.NettyBlockTransferService: Server created on 46454
16/05/24 11:47:57 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/05/24 11:47:57 INFO storage.BlockManagerMasterActor: Registering block manager localhost:46454 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 46454)
16/05/24 11:47:57 INFO storage.BlockManagerMaster: Registered BlockManager
16/05/24 11:47:58 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.

 

I execute the command 

 

 mortagesGraph.vertices.take(5).

 

Error messsage is

 

scala> mortagesGraph.vertices.take(5)
16/05/24 10:08:28 ERROR executor.Executor: Exception in task 0.0 in stage 56.0 (TID 64)
java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Array.newArray(Native Method)
at java.lang.reflect.Array.newInstance(Array.java:70)
at scala.reflect.ClassTag$class.newArray(ClassTag.scala:62)
at scala.reflect.ClassTag$$anon$1.newArray(ClassTag.scala:144)
at org.apache.spark.util.collection.PrimitiveVector.resize(PrimitiveVector.scala:74)
at org.apache.spark.util.collection.PrimitiveVector.$plus$eq(PrimitiveVector.scala:41)
at org.apache.spark.graphx.impl.EdgePartitionBuilder.add(EdgePartitionBuilder.scala:34)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1$$anonfun$apply$1.apply(EdgeRDD.scala:107)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1$$anonfun$apply$1.apply(EdgeRDD.scala:106)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1.apply(EdgeRDD.scala:106)
at org.apache.spark.graphx.EdgeRDD$$anonfun$1.apply(EdgeRDD.scala:104)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

 

 

How can I fix the error?

 

Thanks in advance

 

Carlota Vina