Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark - spark socketexception connection reset by peer

Solved Go to solution

spark - spark socketexception connection reset by peer

Expert Contributor

was trying spark scenario, tried to load csv file of data set around 1M records into a RDD.

did a split by delimiter, and was checking count() which worked.

on the same RDD wanted to check sample data and tried action take(10)which did not work.

Was throwing spark socketexception connection reset by peer

15988-sockect-exception.jpg

Your assistance would be of great help

1 ACCEPTED SOLUTION

Accepted Solutions

Re: spark - spark socketexception connection reset by peer

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

2 REPLIES 2

Re: spark - spark socketexception connection reset by peer

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

Highlighted

Re: spark - spark socketexception connection reset by peer

Expert Contributor

tried creating RDD with collect() and print out using for loop. Was working fine.

Was trying out in pyspark though.

thank you

Don't have an account?
Coming from Hortonworks? Activate your account here