Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark - spark socketexception connection reset by peer

avatar
Super Collaborator

was trying spark scenario, tried to load csv file of data set around 1M records into a RDD.

did a split by delimiter, and was checking count() which worked.

on the same RDD wanted to check sample data and tried action take(10)which did not work.

Was throwing spark socketexception connection reset by peer

15988-sockect-exception.jpg

Your assistance would be of great help

1 ACCEPTED SOLUTION

avatar

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

View solution in original post

2 REPLIES 2

avatar

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

avatar
Super Collaborator

tried creating RDD with collect() and print out using for loop. Was working fine.

Was trying out in pyspark though.

thank you