That's not what it says; it say they just aren't supported, typically because they're not "supported" in Spark either (e.g. experimental API). Supported != doesn't work, just means you can't file a support ticket for it. CDH 5.6 = Spark 1.5 + patches, meaning it's like 1.5.2 likely with a slightly different set of maintenance patches. It might not have unimportant ones that maybe shouldn't be in a maintenance release, or might have a critical one that was created after 1.5.2. Generally speaking there are no other differences; it's just upstream Spark with some tinkering with versions to make it integrate with other Hadoop components correctly. The exception is SparkR, which isn't even shipped, partly because CDH can't ship R itself. ... View more

It includes an implementation of classification using random decision forests. Decision forests actually support both categorical and numeric features. However, for text classification, you're correct that you typically transform your text into numeric vectors via TF-IDF first. This is something you'd have to do separately. Yes, the dimensionality is high. Decision forests can be fine with this, but, they're not the most natural choice for text classification. You may see what I mean that Oryx is not a tool for classification, but a tool for productionizing, which happens to have an implementation of a classifier. In 2.x, you also have an implementation of decision forests, and also don't have magic TF-IDF built in or anything. However the architecture is much more supportive of putting your own Spark-based pipeline and model build into the framework. 1.x did not support this. ... View more

The error points to the problem -- you have perhaps plenty of memory but not enough permgen space in the JVM. Try something like -XX:MaxPermSize=2g in your JVM options to executors ... View more

"Not supported" means you can't file support tickets for it. It's shipped and works though. ... View more

It depends a lot on what you mean by "mature". It is mature in the sense that it's: - a fifth (!) generation architecture - used by real customers in production, yes - not at a beta stage, but at a 2.1.1 release now - built on technologies that Cloudera supports as production ready it's not mature in the sense that it's: - is not itself formally supported by Cloudera -- only a labs project - is still fairly new in its current 2.x form, having finished about 5 months ago I think the architecture is certainly the way to go if you're building this kind of thing on Hadoop. See http://oryx.io. I don't know much about Samoa but I understand it to be a distributed stream-centric ML library. On the plus side, it's probably better than anything in Spark for building huge models incrementally, as this is what it focuses on. On the downside, it doesn't do the model serving element, which Oryx tries to provide, and in a sense Samoa is a much less standard technology than Spark, HDFS, and Kafka. ... View more

Yes, I think that begins to narrow it down. I don't know that you're going to find a big performance difference, since distributions will generally ship the upstream project with only minimal modifications to integrate it. (That said, CDH does let you enable native acceleration for some mathematical operations in Spark MLlib. I don't think other distros enable this and ship the right libraries. It's possible that could matter to your use case.) I'd look at how recent the Spark distribution is. Cloudera ships Spark 1.5 in CDH 5.5; MapR is on 1.4 and Hortonworks on 1.3, with a beta preview of 1.5 at the moment in both cases. We're already integrating the nearly-released Spark 1.6 too. Finally, if you're considering paying for support, I think it bears evaluating how much each vendor invests in Spark. No investment means no expertise and no real ability to fix your problems. At Cloudera, we have a full-time team on Spark, including 4 committers (including me). I think you'll find other vendors virtually non-existent in the Spark community, but, go see for yourself. ... View more

Here, you're using your own build of Spark against an older version of Hive than what's in CDH. That might mostly work, but you're seeing the problems in compiling and running vs different versions. I'm afraid you're on your own if you're rolling your own build, but, I expect you may get much closer if you make a build targeting the same HIve version in CDH. ... View more

I remember some problems with snappy and HBase like this, like somehow an older version used by HBase ended up taking precedence in the app classloader and then it could not quite load properly, as it couldn't see the shared library in the parent classloader. This may be a manifestation of that one. I know there are certainly cases where there is no resolution to the conflict, since an app and Spark may use mutually incompatible versions of a dependency, and one will mess with the other if the Spark and app classloader are connected, no matter what their ordering. For this toy example, you'd just not set the classpath setting since it isn't needed. For your app, if neither combination works, then your options are probably to harmonize library versions with Spark, or shade your copy of the library. ... View more

Replications is an HDFS-level configuration. It isn't something you configure from Spark, and you don't have to worry about it from Spark. AFAIK you set a global replication factor, but can set it per directory too. I think you want to pursue this via HDFS. ... View more

I suppose you can cluster term vectors in V S for this purpose, to discover related terms and thus topics. This is the type of problem where you might more usually use LDA. I know you're using Mahout, but if you ever consider using Spark, there's a chapter on exactly this in our book: http://shop.oreilly.com/product/0636920035091.do ... View more

The output is as you say -- these are the products of the SVD. You can do what you want with them, and it depends on what you're trying to achieve. You can look at the matrix V S to study term similarities, or U S to discover document similarities for example. ... View more

That would be unsupported. I think you'd find support would try to help in this case, but if it legitimately looked like a problem with Spark 1.4, would decline to pursue it. Spark 1.5 is supported in CDH 5.5 of course, coming soon. ... View more

No it is not the same 'because' computation in the paper. The one in the paper is better. However it requires storing a k x k matrix for every user, or computing it on the fly, both of which are pretty prohibitive. They're not hard to implement though. This is a cheap-o non-personalized computation based on item similarity.   No, the system does not serve the original data, just results from the factored model. It's assumed that, if the caller needs this info, the caller has it, and its purpose is generally not specific to the core recommender, so accessing this data is not part of the engine. ... View more

That's fine. The machine needs to be able to communicate with the cluster of course. Usually you would make the Hadoop configuration visible as well and point to it with HADOOP_CONF_DIR. I think that will be required to get MapReduce to work. ... View more