Created 03-28-2016 05:39 AM
Has anyone tried to create a local spark context within a Storm bolt to load a saved Spark model instead of creating a model using exported weights or PMML? It seems that there is a Log4J dependency conflict between Storm and Spark.
Created 03-29-2016 06:02 PM
Just looked through the Metron project and none of the POM files seem to have a reference to Spark. I bet they are either using PMML or exporting weights. I did a bit more reading as well and the more I think about it the more it seems like that pattern is just not such a great idea. Thanks for your input.
Created 03-28-2016 01:53 PM
Have you seen this thread Vadim? https://community.hortonworks.com/questions/24092/how-to-use-spark-mllib-model-in-storm.html
Also if you're having dependency issues you can use exclude tag in pom.XML to resolve those issues. https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html
Created 03-28-2016 05:10 PM
Yes, and that is how I am applying my models to the demos that I have built so far. I was interested in whether it is possible to create a Spark context in a Storm Bolt. Sounds like the answer might be no. Is it?
Created 03-29-2016 04:54 PM
@Vadim checkout the Metron project, they are doing spark model scoring in storm if i remember correctlyh from SKO https://github.com/apache/incubator-metron
Created 03-29-2016 06:02 PM
Just looked through the Metron project and none of the POM files seem to have a reference to Spark. I bet they are either using PMML or exporting weights. I did a bit more reading as well and the more I think about it the more it seems like that pattern is just not such a great idea. Thanks for your input.