I reproduced this. Everything else works fine, you can see the model generates a MAP of about 0.15 on Hadoop. It's just the last step where it seems to incorrectly decide the rank is insufficient. There is always a bit of heuristic here; nothing is ever literally "singular" due to machine precision. So it could be a case of loosening or improving the heuristic. I'll have to debug a little more. ... View more

That's a weird one indeed. NoSuchMethodError generally means you have built your app against a different version of a library than you run against.   CDH 5.2 contains Spark 1.1.0 plus a few critical upstream fixes. None of those changes should ever affect source or binary compatibility. You can always build against the exact CDH 5.2 artifacts anyway, but it shouldn't matter. 1.1.0, 1.1.1, is all the same from an API perspective.   But, here, there should be no incompatibility at all anyway, since you should not be bundling Spark libs (or Hadoop libs) with your app. Are you marking them as 'provided' dependencies? ... View more

You can ignore the native libraries message. It doesn't affect anything. Right, X and Y are deleted after. It may be hard to view them before that happens. The hash from IDs to ints is a consistent one, so the same string will always map to the same ID.   Something funny is going on here and it's probably subtle but simple, like an issue with how the data is read. Your comment about the IDs kind of suggests that the data files aren't being read as intended, so maybe all of these IDs are being treated quite differently as if they are unrelated. That could somehow explain poor performance and virtually 0 rank -- which should all but impossible with so much data and a reasonable default rank of <100.   Is it possible to send me a link to the data privately, and your config? I can take a look locally. ... View more

Although the result can vary a bit randomly from run to run, and it's possible you're on the border of insufficient rank, it sounds like this happens consistently?   Are there any errors from the Hadoop workers? do X/ and Y/ contain data? It sounds like the process has stopped too early. I suppose double-check that you do have the same data on HDFS. The config is otherwise the same? ... View more

Hadoop still has config files for sure. They can end up wherever you want them to. I though they're still at $HADOOP_HOME/conf in the vanilla Hadoop tarball, but I took a look at 2.5.2 and it's at$HADOOP_HOME/etc/hadoop in fact. In any event if they're at /usr/local/hadoop/etc/hadoop in your installation, then that's what you set $HADOOP_CONF_DIR to. Just wherever they really are. This is one of Hadoop's standard environment variables. If you're up and running then this is working.   Yes that sounds like about what you do to install snappy. They are libs that should be present on the cluster machines. ... View more

Caused by: java.lang.IllegalStateException: Not a directory: /etc/hadoop/conf   Is HADOOP_CONF_DIR set and set to /usr/local/hadoop ? that's what it's complaining about, that it can't find Hadoop config in a default location. ... View more

Spark defaults to run with a local master IIRC. You should set "--master yarn-client" to actually use YARN. I assume it's not different for pyspark vs spark-shell. ... View more