Support Questions
Find answers, ask questions, and share your expertise

spark - spark socketexception connection reset by peer

Expert Contributor

was trying spark scenario, tried to load csv file of data set around 1M records into a RDD.

did a split by delimiter, and was checking count() which worked.

on the same RDD wanted to check sample data and tried action take(10)which did not work.

Was throwing spark socketexception connection reset by peer

15988-sockect-exception.jpg

Your assistance would be of great help

1 ACCEPTED SOLUTION

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

View solution in original post

2 REPLIES 2

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

Expert Contributor

tried creating RDD with collect() and print out using for loop. Was working fine.

Was trying out in pyspark though.

thank you

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.