It means basically what it says, that you're writing some program that accesses /user/spark but you're not running as spark, the user that can access that directory. ... View more

You're running on YARN, so you should see the application as a "FAILED" application in the Resource Manager UI. Click through and you can find the logs of individual containers, which should show some failure. ... View more

Yes, you shouldn't be able to run this as a stand-alone app. Hm, try putting the jar file last? that is how the script says to do it. ... View more

You need <scope>provided</scope> as well. ... View more

How are you executing this? it sounds like you may not be using spark-submit, or, are accidentally bundling Spark (perhaps a slightly different version) into your app. Spark deps should be 'provided' in your build and you'll want to use spark-submit to submit. You don't set master in your SparkConf in code. ... View more

Hm, what do you mean by commutative and associative? and do you mean Hadoop clusters?   I'm not sure there's a particular limit to what a Hadoop cluster can do well other than that it's fundamentally a data-parallel paradigm. But most things can be done efficiently in this paradigm, especially random forests. The only things that don't work well are things that require extremely fast async communication -- MPI-style computations.   Decision forests are strong and you can certainly do them well on Hadoop. ... View more

Random decision forests in MLlib 1.2 can do classification or regression. Yes it can do bagging. I don't believe it's by feature, no. ... View more

MLlib supports SVMs in Spark 1.1. It supports Decision Trees in 1.1, and Decision Forests in 1.2, which is not quite yet released.   Mahout has an implementation of SVMs and Decision Forests. They are both fairly old and MapReduce-based. ... View more

Hm, why not just use the Spark that is part of CDH? If you want 1.1, can you update to CDH 5.2? Are there more logs? this isn't the underlying error. ... View more