Support Questions
Find answers, ask questions, and share your expertise

Kryo serialization failed: Buffer overflow.

New Contributor

I am getting the org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow when I am execute the collect on 1 GB of RDD(for example : My1GBRDD.collect).

When I am execution the same thing on small Rdd(600MB), It will execute successfully. The problem with above 1GB RDD.

For more details please refer the following steps which I do.

1. Create RDD of input file.

2. mapToPair on the RDD.

3. groupByKey() on the RDD.

4. collectAsMap on the RDD.

 

On the 4th step I got the SparkException as follows,

org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 37
Serialization trace:
otherElements (org.apache.spark.util.collection.CompactBuffer). To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:350)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:393)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 37

 

1 REPLY 1

Explorer

Hi , 

How did you solve this issue , i have the same.