Support Questions

Find answers, ask questions, and share your expertise

Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?



I need to save a model in python spark 1.6.0. I know save()/load functions are available in 2.0 but I'm not in a position to upgrade our HDP cluster at this current time and need a hack.

I know Scala 1.6 does support saving of models. Is there some way I could share my model object from python to scala. Im using zeppelin so this might work between paragraphs?

Any help much appreciated



ps - i tried pickling the model object but that didnt work

Expert Contributor

To understand, if you want to save the model to HDFS, then you can use the functionality. The article below is an interested walk through the entire model process:

You can also save the model into PMML format:

Cloudera Employee

I have checked spark 1.5.0 documentation and,"hdfs path"), <ModelClass>.load(sc,"hdfs path") are supported. Can you give a specific example ?