Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is the reason behind Spark Functions extends Serializable.?

Solved Go to solution
Highlighted

What is the reason behind Spark Functions extends Serializable.?

Explorer

I want to get exact reason behind having functions serializable in Spark and want to know the if possible want to know the scenarios, where can be issues because of Serialization,

 

As far as my understanding goes, to ensure seam less no side-effect parallel processing, instead of sending the data liike imperative paradigm,

function will be sent to the node and data gets processed parallely.

 

Is my above thought Correct.??? As far as my study, Functional programming is a very good way forward for parallel processing/concurrent programming, so i thought this is the reason.

 

As we are passing function, is it the security reason behind having functions serializable.?

 

Thanks In advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: What is the reason behind Spark Functions extends Serializable.?

Master Collaborator

I don't think it has to do with functional programming per se, but yes, it's because the function/code being executed has to be sent from the driver to the executors, and so the function object itself must be serializable. It has no relation to security.

View solution in original post

1 REPLY 1
Highlighted

Re: What is the reason behind Spark Functions extends Serializable.?

Master Collaborator

I don't think it has to do with functional programming per se, but yes, it's because the function/code being executed has to be sent from the driver to the executors, and so the function object itself must be serializable. It has no relation to security.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here