Support Questions
Find answers, ask questions, and share your expertise

Does HDP spark include netLib optimizations?

Hi,

I wonder if the spark version provided by HDP will actually provide http://spark.apache.org/docs/latest/ml-guide.html mentioned optimizations compiled in:

MLlib uses the linear algebra package Breeze, which depends on netlib-java for optimised numerical processing. If native libraries1 are not available at runtime, you will see a warning message and a pure JVM implementation will be used instead.
5 REPLIES 5

There is a similar question: https://community.hortonworks.com/questions/52359/netlib-java-and-anaconda-for-spark-ml.html without netlike lgpl answered.

@Georg Heiler

This is not about HDP Spark including netLib optimizations. Not even Apache Spark include that.

https://issues.apache.org/jira/browse/SPARK-19423, last comment.

This is a netlib-java question. You could enable that yourself. @Bernhard Walter believes that is already there (?)

+++

If this helped, please vote/accept answer.

Well, at least spark has the option to include the machine learning options. I am just interested if hdp is deploying these optimizations or if manual work is required.

@Georg Heiler

It is not part of Apache Spark and sub-sequently not part of HDP included Spark. Check this link to understand manual actions for your OS specifics: https://github.com/fommil/netlib-java

+++

If this helped, please vote/accept answer.

I see some hdp will only ship default spark? I mean it is just a compile switch for spark. Having that would be nice. Can I just replace the hdp binaries?

; ;