Sean,   (1)  How Oryx serving layer knows there is new model generation? I traced the codes and it seems continues polling to check the generation status from folder. And, the delay for polling seems different from "no available generation" status (different minutes)...   Is that right ? Can you explain a little bit?   (2) When the new generation is replacing the old generation, I see a variable "recentlyActiveUsers"... What's that used for ? https://github.com/cloudera/oryx/blob/85001d5fce34d66c6d9179ab6e82e1cecb3b17ee/als-serving/src/main/java/com/cloudera/oryx/als/serving/generation/GenerationLoader.java#L116   (3) Question on "loadModel(int generationID, Generation currentGeneration)" ? How the model of old generation is replaced by new generation? https://github.com/cloudera/oryx/blob/85001d5fce34d66c6d9179ab6e82e1cecb3b17ee/als-serving/src/main/java/com/cloudera/oryx/als/serving/generation/GenerationLoader.java#L67 Originally, I am thinking new generation is loaded from disk to memory and then the old-generation in the memory is replaced by loaded new generation. However, after tracing the codes, it seems there is no such "hot-swap". Based on my tracing and assume currentGeneration=2 and generationID=3 (that's, currentGeneration is going to be replaced by 3). It seems (step1) the new model is saved to currentGeneration and (step2) the old entries left in currentGeneration are removed. If such a case, it looks possible there is an "intermediate" model mixing old and new generation. Will this "intermediate" model be queried by the recommender ? Can you confirm or explain a little bit ?     Thanks.     ... View more

Sean,   As I posted in other discussion threads, we are trying to run Oryx 1.0 with CDH5.4.1.   One thing I noticed is that when computing with Hadoop, we have 7.5 million users in training set. However, after the training , it generates X matrix of about 7.3 million users.   I checked the log and not found any message/error related to this. I also tried the same training dataset in the local computation (single VM), and it gets 7.5 million users in X fine.  I checked those users got lost during Hadoop computation and noticed that all their preference values are less than 0.01... I think it's definitely making sense to ignore association of very low preference value. But, I cannot find such "config" or "control"... Is there is such thing in Oryx ? Why it's filtering with Hadoop computation, but not in single VM computation ?   Thanks. ... View more