Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

avatar
Expert Contributor

Hi,

I need to save a model in python spark 1.6.0. I know save()/load functions are available in 2.0 but I'm not in a position to upgrade our HDP cluster at this current time and need a hack.

I know Scala 1.6 does support saving of models. Is there some way I could share my model object from python to scala. Im using zeppelin so this might work between paragraphs?

Any help much appreciated

3 REPLIES 3

avatar
Expert Contributor

ps - i tried pickling the model object but that didnt work

avatar
Expert Contributor

To understand, if you want to save the model to HDFS, then you can use the model.save functionality. The article below is an interested walk through the entire model process:

https://www.codementor.io/spark/tutorial/building-a-recommender-with-apache-spark-python-example-app...

You can also save the model into PMML format:

https://spark.apache.org/docs/1.6.0/mllib-pmml-model-export.html

avatar
Contributor

I have checked spark 1.5.0 documentation and model.save(sc,"hdfs path"), <ModelClass>.load(sc,"hdfs path") are supported. Can you give a specific example ?