Member since
12-07-2016
10
Posts
0
Kudos Received
0
Solutions
03-02-2017
03:07 PM
Task Not serializable is unrelated and very common. The way the scala API works, operations on RDDs like map() work by having the state of the lambda expression copied over to all the worker nodes and then executed. For this to happen, every object referenced inside the expression must be "Serializable", in the strict java API way: it is declared as something which can be serialized to a byte stream, sent over the network and reconstructed at the far end. Something you have declared outside the map, which you are trying to use it inside, isn't serializable. At a guess: one of the Jetty classes, like the "exchange" variable. Workaround? Create the object inside the lambda expression, out of data that has been serialized (strings etc)
... View more
02-27-2017
07:18 AM
How can see the dependency information needed to solve my error?
... View more
02-26-2017
11:25 AM
thanks for replies. could I do it like followings?
export PYSPARK_PYTHON=/opt/anaconda/bin/python spark-submit spark-job.py
... View more
12-07-2016
02:51 AM
from root user I submit 'su hdfs', then t would not ask to me. Thank u!!
... View more