question spark - spark socketexception connection reset by peer in Archives of Support Questions (Read Only)

spark - spark socketexception connection reset by peer

sreeviswa_athic — Sun, 18 Aug 2019 06:22:00 GMT

was trying spark scenario, tried to load csv file of data set around 1M records into a RDD.

did a split by delimiter, and was checking count() which worked.

on the same RDD wanted to check sample data and tried action take(10)which did not work.

Was throwing spark socketexception connection reset by peer

Your assistance would be of great help

Re: spark - spark socketexception connection reset by peer

dineshc — Mon, 05 Jun 2017 22:32:39 GMT

If you are using PySpark, there appears to be a bug where pyspark crashes for large datasets.

https://issues.apache.org/jira/browse/SPARK-12261

Since you are just trying to see sample data, you could use collect and then print.

However, collect should not be used for large datasets as it brings all the data to driver node and could basically make the driver node run out of memory.

This link gives a detail on how to print the rdd elements using Scala.

Refer here for PySpark.

Re: spark - spark socketexception connection reset by peer

sreeviswa_athic — Tue, 06 Jun 2017 01:06:01 GMT

tried creating RDD with collect() and print out using for loop. Was working fine.

Was trying out in pyspark though.

thank you