I figured out the problem here.   Most of the Object i used in my project is complex. Ofcourse all of them implements 'Serializable' interface. In this case, the Spark engine is supposed to use 'JavaSerializer'(org.apache.spark.serializer.JavaSerializer), but by default it is using 'KryoSerializer'(org.apache.spark.serializer.KryoSerializer).   Solution: 1. Go to Cloudera manager 2. Select 'spark2' service 3. Go to configuration tab 4. search for 'spark.serializer' property and change it to 'org.apache.spark.serializer.JavaSerializer', where as the default serializer is 'org.apache.spark.serializer.KryoSerializer' 5. Dont need to restart Spark2, just make sure 'client configuration' is deployed. 6. Restart livy. (optional, but better do that to close current applications) ... View more