spark - spark socketexception connection reset by peer

[#1]

was trying spark scenario, tried to load csv file of data set around 1M records into a RDD.

did a split by delimiter, and was checking count() which worked.

on the same RDD wanted to check sample data and tried action take(10)which did not work.

Was throwing spark socketexception connection reset by peer

Your assistance would be of great help

[#2 · Accepted solution]

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

[#3]

tried creating RDD with collect() and print out using for loop. Was working fine.

Was trying out in pyspark though.

thank you