Apologies, I'm mixing up 1.x and 2.x. The default evaluation metric in 1.x is mean average precision, or MAP. This is a measure of how much the top recommendations contained some items that were held out for the user. In local mode you can find lines like "Mean average precision: xxx" in the logs.   In distributed mode, now that I review the code, I don't see that it is ever logged. It is written to a file called "MAP" under the subdirectory for the iteration. I can make the mapper workers output their own local value of MAP at least.   In 2.x the metric is AUC, which is basically a measure of how likely it is that a 'good' recommendation (from the held out data set) ranks above a random item. It is a broader, different measure. This you should find printed in the logs if you're using 2.x for sure, along with hyperparams that yielded it. ... View more

It depends a lot on just what you mean by 'analysis'. 1 machine could be just fine. In general I think you will want to play with Spark, and Spark loves memory. So 8GB RAM seems a bit small, but 4 cores is OK, and I bet you have plenty of disk space.   I do not agree at all that you need 10TB of disk space. That is orders of magnitude overkill for a 10GB data set. ... View more

Yes they will usually be in [0,1] but not always. They aren't probabilities. They are entries in X*Y', yes. I think it's safe to take values >=1 as a very strong positive. What's a good cutoff? it really depends on your semantics and use case. They are comparable across models so you can probably determine a value empirically with some testing. ... View more

Yes, I meant the server side. Actually, looking at the source, it should allow up to 65536 bytes of header data. ... View more

That could suggest that the amount of resource that is available to your Spark jobs is not big enough to accommodate the resources that Talend or your app are requesting. I don't know whether you mean only 2 cores are available or 2 are requested, but the question is whether the request exceeds what's available. I'd check on this aspect. For example if running via YARN, see how much resource YARN can allocate and look at your logs to see what Spark thinks it's asking for. ... View more