Hi All We are trying to make a simple map-reduce task fom some mongo documents to be inserted into hdfs simplified on plain records, the origin of data is about 230k documents that weights about 2.5Gb of data, basically what we do is convert that documents on 20mill lines to insert on csv. We tested with half records and it runs so I think it can have a resource problem. The result is near the same run locally than in yarn, locally we can see: 18/09/14 06:42:18 ERROR PythonRDD: Error while sending iterator java.net.SocketException: Connection reset the command sometimes is: spark-submit --driver-memory 8G --executor-memory 12G --executor-cores 6 --conf spark.driver.maxResultSize="4G" --conf spark.default.parallelism=100 --conf spark.sql.shuffle.partitions=100 combine_test.py but we have tested with different partitions and different memory numbers with the same results. We have three vms with 6cores and 24Gb ram each, and a fourth one as history server. Do you think whe can tune the GC settings to solve this? Thanks in advance Roberto. UPDATE The error comes after a map function and during a reduce
code (simplified) def mappedSearchsResults(obj):
correlationId, values = obj
pair = []
search = None
result = None
for item in values:
if 'contentType' in item:
if (item['contentType']=='Request'):
decodeditem = item['fullRequest'].decode("base64")
search = json.loads(decodeditem)
elif (item['contentType']=='Response'):
decodeditem = item['fullResponse'].decode("base64")
result = json.loads(decodeditem)
else:
if isSearch(item):
search = item
elif isResults(item):
result = item
if (search != None and result != None):
tupleSearchResult = (search, result)
pair.append(tupleSearchResult)
search = None
result = None
return pair
httpRequestsSource = httpRequestsSource.map(mappedSearchsResults)
httpRequestsSource = httpRequestsSource.reduce(lambda x,y: x+y) It crashes during the reduce ERROR PythonRDD: Error while sending iterator
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:192)
at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:204)
at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:204)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:204)
at org.apache.spark.api.python.PythonRDD$$anon$1$$anonfun$run$1.apply$mcV$sp(PythonRDD.scala:400)
at org.apache.spark.api.python.PythonRDD$$anon$1$$anonfun$run$1.apply(PythonRDD.scala:400)
at org.apache.spark.api.python.PythonRDD$$anon$1$$anonfun$run$1.apply(PythonRDD.scala:400)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.api.python.PythonRDD$$anon$1.run(PythonRDD.scala:401)
I have read this similar issue but I do not understand the "pass" solution: https://issues.apache.org/jira/browse/SPARK-12261
... View more
