That's not what it says; it say they just aren't supported, typically because they're not "supported" in Spark either (e.g. experimental API). Supported != doesn't work, just means you can't file a support ticket for it. CDH 5.6 = Spark 1.5 + patches, meaning it's like 1.5.2 likely with a slightly different set of maintenance patches. It might not have unimportant ones that maybe shouldn't be in a maintenance release, or might have a critical one that was created after 1.5.2. Generally speaking there are no other differences; it's just upstream Spark with some tinkering with versions to make it integrate with other Hadoop components correctly. The exception is SparkR, which isn't even shipped, partly because CDH can't ship R itself. ... View more

It includes an implementation of classification using random decision forests. Decision forests actually support both categorical and numeric features. However, for text classification, you're correct that you typically transform your text into numeric vectors via TF-IDF first. This is something you'd have to do separately. Yes, the dimensionality is high. Decision forests can be fine with this, but, they're not the most natural choice for text classification. You may see what I mean that Oryx is not a tool for classification, but a tool for productionizing, which happens to have an implementation of a classifier. In 2.x, you also have an implementation of decision forests, and also don't have magic TF-IDF built in or anything. However the architecture is much more supportive of putting your own Spark-based pipeline and model build into the framework. 1.x did not support this. ... View more

The error points to the problem -- you have perhaps plenty of memory but not enough permgen space in the JVM. Try something like -XX:MaxPermSize=2g in your JVM options to executors ... View more

"Not supported" means you can't file support tickets for it. It's shipped and works though. ... View more

It depends a lot on what you mean by "mature". It is mature in the sense that it's: - a fifth (!) generation architecture - used by real customers in production, yes - not at a beta stage, but at a 2.1.1 release now - built on technologies that Cloudera supports as production ready it's not mature in the sense that it's: - is not itself formally supported by Cloudera -- only a labs project - is still fairly new in its current 2.x form, having finished about 5 months ago I think the architecture is certainly the way to go if you're building this kind of thing on Hadoop. See http://oryx.io. I don't know much about Samoa but I understand it to be a distributed stream-centric ML library. On the plus side, it's probably better than anything in Spark for building huge models incrementally, as this is what it focuses on. On the downside, it doesn't do the model serving element, which Oryx tries to provide, and in a sense Samoa is a much less standard technology than Spark, HDFS, and Kafka. ... View more

Yes, I think that begins to narrow it down. I don't know that you're going to find a big performance difference, since distributions will generally ship the upstream project with only minimal modifications to integrate it. (That said, CDH does let you enable native acceleration for some mathematical operations in Spark MLlib. I don't think other distros enable this and ship the right libraries. It's possible that could matter to your use case.) I'd look at how recent the Spark distribution is. Cloudera ships Spark 1.5 in CDH 5.5; MapR is on 1.4 and Hortonworks on 1.3, with a beta preview of 1.5 at the moment in both cases. We're already integrating the nearly-released Spark 1.6 too. Finally, if you're considering paying for support, I think it bears evaluating how much each vendor invests in Spark. No investment means no expertise and no real ability to fix your problems. At Cloudera, we have a full-time team on Spark, including 4 committers (including me). I think you'll find other vendors virtually non-existent in the Spark community, but, go see for yourself. ... View more