Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

Contributor

Hi,

I need to save a model in python spark 1.6.0. I know save()/load functions are available in 2.0 but I'm not in a position to upgrade our HDP cluster at this current time and need a hack.

I know Scala 1.6 does support saving of models. Is there some way I could share my model object from python to scala. Im using zeppelin so this might work between paragraphs?

Any help much appreciated

3 REPLIES 3

Re: Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

Contributor

ps - i tried pickling the model object but that didnt work

Re: Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

Expert Contributor

To understand, if you want to save the model to HDFS, then you can use the model.save functionality. The article below is an interested walk through the entire model process:

https://www.codementor.io/spark/tutorial/building-a-recommender-with-apache-spark-python-example-app...

You can also save the model into PMML format:

https://spark.apache.org/docs/1.6.0/mllib-pmml-model-export.html

Highlighted

Re: Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

New Contributor

I have checked spark 1.5.0 documentation and model.save(sc,"hdfs path"), <ModelClass>.load(sc,"hdfs path") are supported. Can you give a specific example ?