I'm not suggesting you log in as spark or a superuser. You shouldn't do this. Instead, change your app to not access directories you don't have access to as your user. ... View more

Yes I know what commutativity and associativity are, I was wondering how it related to Hadoop and decision forests. In theory a reduce function should be commutative and associative, but in practice it does not need to be in MapReduce, and a MapReduce as a unit is not, and certainly Spark is not. There is no practical computation paradigm limitation of this form.   I looked into the MLlib RDF code and it does look like it selects features too at random, depending on the configuration. So you could say it bags by examples and features.   The oryx implementation also certainly does all of what you describe. https://github.com/cloudera/oryx/tree/master/rdf-computation ... View more

By default the distributed implementations will prune infrequent items and low similarity to scale up. With a tiny data set, this can mean some items are removed entirely, or become un-recommendable. While you can change this behavior, it is not useful to run the Hadoop-based job on such a small data set. ... View more

There is no user-based recommender based on Hadoop MapReduce. The closest thing is indeed an item-based implementation in https://github.com/apache/mahout/tree/master/mrlegacy/src/main/java/org/apache/mahout/cf/taste/hadoop/item  It is also a recommender, but no it is always using item similarity. ... View more