What is the reason behind Spark Functions extends Serializable.?

Srini_D — Fri, 16 Sep 2022 09:37:06 GMT

I want to get exact reason behind having functions serializable in Spark and want to know the if possible want to know the scenarios, where can be issues because of Serialization,

As far as my understanding goes, to ensure seam less no side-effect parallel processing, instead of sending the data liike imperative paradigm,

function will be sent to the node and data gets processed parallely.

Is my above thought Correct.??? As far as my study, Functional programming is a very good way forward for parallel processing/concurrent programming, so i thought this is the reason.

As we are passing function, is it the security reason behind having functions serializable.?

Thanks In advance.

Re: What is the reason behind Spark Functions extends Serializable.?

srowen — Thu, 06 Aug 2015 09:36:39 GMT

I don't think it has to do with functional programming per se, but yes, it's because the function/code being executed has to be sent from the driver to the executors, and so the function object itself must be serializable. It has no relation to security.

question What is the reason behind Spark Functions extends Serializable.? in Archives of Support Questions (Read Only)

What is the reason behind Spark Functions extends Serializable.?

Re: What is the reason behind Spark Functions extends Serializable.?