question Re: How to Use Spark MLLib Model in Storm? in Support Questions

How to Use Spark MLLib Model in Storm?

yjiang — Tue, 22 Mar 2016 22:58:48 GMT

Is there a way to train the model offline in Spark MLLib, and then use it for online ML in Storm?

Re: How to Use Spark MLLib Model in Storm?

hkropp — Tue, 22 Mar 2016 23:14:50 GMT

You can use PMML (https://de.wikipedia.org/wiki/Predictive_Model_Markup_Language).

Spark does support (not all) model to be exported to PMML:

http://spark.apache.org/docs/latest/mllib-pmml-model-export.html

(UPDATE: As @Simon Elliston Ball rightfully points out in his answer, in case the PMML model is not supported the Spark libs can be reused as most of them have no dependency to the SparkContext)

One way could be to use JPMML with Java in Storm:

http://henning.kropponline.de/2015/09/06/jpmml-example-random-forest/

https://github.com/jpmml/jpmml-storm

The other could be to use R in Storm. I have seen it done, but don't have a reference at hand.

Re: How to Use Spark MLLib Model in Storm?

hkropp — Tue, 22 Mar 2016 23:17:51 GMT

In in an advanced architecture you would leverage Zookeeper to announce a new model to the topology without taking it offline.

Re: How to Use Spark MLLib Model in Storm?

sball — Wed, 23 Mar 2016 00:08:34 GMT

PMML is certainly a good option, but be aware that Spark does not support the transformation elements of PMML, so you will need to recreate any feature scaling and transformation before the scoring step.

The other thing to note is that many of the Spark Model classes do not depend on the spark context, so you can link spark to you storm topology and just use the Spark Model itself.

This can lead to some unnecessary code in your jar, but has the advantage that you don't need to go through the PMML format.

Re: How to Use Spark MLLib Model in Storm?

hkropp — Wed, 23 Mar 2016 02:22:58 GMT

+1 for the aspect to reuse Spark code itself