Support Questions
Find answers, ask questions, and share your expertise

Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

Contributor

Hi,

I need to save a model in python spark 1.6.0. I know save()/load functions are available in 2.0 but I'm not in a position to upgrade our HDP cluster at this current time and need a hack.

I know Scala 1.6 does support saving of models. Is there some way I could share my model object from python to scala. Im using zeppelin so this might work between paragraphs?

Any help much appreciated

3 REPLIES 3

Contributor

ps - i tried pickling the model object but that didnt work

Expert Contributor

To understand, if you want to save the model to HDFS, then you can use the model.save functionality. The article below is an interested walk through the entire model process:

https://www.codementor.io/spark/tutorial/building-a-recommender-with-apache-spark-python-example-app...

You can also save the model into PMML format:

https://spark.apache.org/docs/1.6.0/mllib-pmml-model-export.html

Cloudera Employee

I have checked spark 1.5.0 documentation and model.save(sc,"hdfs path"), <ModelClass>.load(sc,"hdfs path") are supported. Can you give a specific example ?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.