That's really what IDRescorer is for, yes. If you need it in distributed mode you can reimplement the same idea by changing the code. I don't think it's really a clustering problem; you're just filtering based on clear attributes. You could also think of it a search relevance problem, and combine the results of a recommender and search engine in your app. No, ALS has no concept of attributes. It's a different, longer story, but you can always use 'fake' users and items corresponding to topics or labels to inject this information in the ALS model. ... View more

You are using standalone mode, i.e. the "Spark" service and not YARN? Check to see that the workers are running and healthy. Did executors register at startup? Double-check they have the memory you think. If not, they may not be accepting work because they do not have the memory to allocate that you expect. ... View more

Yes, that's all correct. Set time-threshold to 24 hours (1440 minutes) to rebuild once a day, regardless of the amount of data that has been written. Yes, the amount of data used to build the model is always increasing (unless you are manually deleting data, or, decaying data). It does sum up all counts for each user-item pair, so it is somewhat compacted this way. ... View more

Scala is the native language of Spark. All else equal, it will be easiest to use Spark in Scala. However, of course, not everyone knows Scala or is using it in other projects.   Using it from Java is only slightly less convenient. You will write more code since Java's handling of anonymous classes is quite verbose before Java 8. All of the Scala APIs can be called from Java too, although some look weird when accessed from Java. Most APIs have a Java-friendlier version where necessary to ease this integration.   Python is probably the least easy to use since it is not JVM-based. There is a runtime overhead to translating back and forth between Spark and Python. Not all APIs are 'translated' to Python. Still, it works, and is useful if, well, you know Python and want to use it. ... View more

Yes, that's it.   Can you delete the model while it's running? no, it would leave the instance unable to serve anything, so I hadn't seen the point of that versus just shutting down the server.   No you can't rebuild with different params without restarting, although the computation layer is the thing that rebuilds the model. The serving layer just loads what it is given.   In the next version (the 2.x rewrite) it will try to find the best params automatically anyway. ... View more