Is there a way to save a model in PYSPARK (python) 1.6.0 to HDFS ?

[#1]

Hi,

I need to save a model in python spark 1.6.0. I know save()/load functions are available in 2.0 but I'm not in a position to upgrade our HDP cluster at this current time and need a hack.

I know Scala 1.6 does support saving of models. Is there some way I could share my model object from python to scala. Im using zeppelin so this might work between paragraphs?

Any help much appreciated

[#2]

ps - i tried pickling the model object but that didnt work

[#3]

To understand, if you want to save the model to HDFS, then you can use the model.save functionality. The article below is an interested walk through the entire model process:

https://www.codementor.io/spark/tutorial/building-a-recommender-with-apache-spark-python-example-app...

You can also save the model into PMML format:

https://spark.apache.org/docs/1.6.0/mllib-pmml-model-export.html

[#4]

I have checked spark 1.5.0 documentation and model.save(sc,"hdfs path"), <ModelClass>.load(sc,"hdfs path") are supported. Can you give a specific example ?