Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

What is the reason behind Spark Functions extends Serializable.?

avatar
Rising Star

I want to get exact reason behind having functions serializable in Spark and want to know the if possible want to know the scenarios, where can be issues because of Serialization,

 

As far as my understanding goes, to ensure seam less no side-effect parallel processing, instead of sending the data liike imperative paradigm,

function will be sent to the node and data gets processed parallely.

 

Is my above thought Correct.??? As far as my study, Functional programming is a very good way forward for parallel processing/concurrent programming, so i thought this is the reason.

 

As we are passing function, is it the security reason behind having functions serializable.?

 

Thanks In advance.

1 ACCEPTED SOLUTION

avatar
Master Collaborator

I don't think it has to do with functional programming per se, but yes, it's because the function/code being executed has to be sent from the driver to the executors, and so the function object itself must be serializable. It has no relation to security.

View solution in original post

1 REPLY 1

avatar
Master Collaborator

I don't think it has to do with functional programming per se, but yes, it's because the function/code being executed has to be sent from the driver to the executors, and so the function object itself must be serializable. It has no relation to security.