question Re: Oryx ALS: X and Y do not have sufficient rank in Support Questions

Oryx ALS: X and Y do not have sufficient rank

JamesConner — Fri, 16 Sep 2022 15:39:27 GMT

A little info on the system this is running on:

I'm running CDH5 Beta1 on RHEL6U5, using the parcel installation method. I've set $JAVA_HOME to the cloudera installed 1.7_25 version. Oryx was downloaded from github and built from source, using the hadoop22 profile. The data source for the ALS job is on HDFS, not local.

I have a dataset containing 3,766,950 observations in User,Product,Strength format, which I am trying to use with the Oryx ALS collaborative filtering algorithm. Roughly 67.37% of the observations have a weight of 1. My problem is that when attempting to execute the ALS job, the results are that either X or Y does not have sufficient rank, and are thus deleted.

I've attempted running the Myrrix ParameterOptimizer using the following command (3 steps, 50% sample):

java -Xmx4g -cp myrrix-serving-1.0.1.jar net.myrrix.online.eval.ParameterOptimizer data 3 .5 model.features=10:150 model.als.lambda=0.0001:1

It recommended using {model.als.lambda=1, model.features=45}, which I then used in the configuration file.

The configuration file itself is very simple:

model=${als-model}
model.instance-dir=/Oryx/data
model.local-computation=false
model.local-data=false
model.features=45
model.lambda=1
serving-layer.api.port=8093
computation-layer.api.port=8094

And the computation command:

java -Dconfig.file=als.conf -jar computation/target/oryx-computation-0.4.0-SNAPSHOT.jar

After 20m or so of processing, this is the final few lines of output:

Thu Feb 06 12:49:08 EST 2014 INFO Loading X and Y to test whether they have sufficient rank
Thu Feb 06 12:49:24 EST 2014 INFO Matrix is not yet proved to be non-singular, continuing to load...
Thu Feb 06 12:49:24 EST 2014 WARNING X or Y does not have sufficient rank; deleting this model and its results
Thu Feb 06 12:49:24 EST 2014 INFO Deleting recursively: hdfs://nameservice1/Oryx/data/00000/X
Thu Feb 06 12:49:24 EST 2014 INFO Deleting recursively: hdfs://nameservice1/Oryx/data/00000/Y
Thu Feb 06 12:49:24 EST 2014 INFO Signaling completion of generation 0
Thu Feb 06 12:49:24 EST 2014 INFO Deleting recursively: hdfs://nameservice1/Oryx/data/00000/tmp
Thu Feb 06 12:49:24 EST 2014 INFO Dumping some stats on generation 0
Thu Feb 06 12:49:24 EST 2014 INFO Generation 0 complete

Any ideas on why this isn't working with using the recommended Features count and Lambda? The ALS audioscrobbler example works fine, and the data format is similar (though the strengths are considerably smaller on my dataset).

Thanks in advance,

James

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Thu, 06 Feb 2014 18:34:27 GMT

Hmm that does sound strange. lambda = 1 is on the high side, although it may have come out as the best value given the values tested in the optimizer and given the variation in random starting points, etc. There is some randomness on the other side too when it builds and tests the factorization.

My first guess is: decrease lambda. You might re-run the optimizer and restrict it to at most 0.1. This isn't a great answer but think it may be the fastest path to something working.

Longer-term, this is going to be rewritten to fully integrate parameter search into model building, so it won't be this separate and maybe disagreeing process.

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Thu, 06 Feb 2014 19:33:23 GMT

Hi Sean,

I should have mentioned that I've tried a few variations, each resulting in the same error each time. I've tried the following combinations so far, each with the same result as when I followed the recommended Feature/Lambda settings:

Features : Lambda

20 : 0.065

100 : 0.065

45 : 1

45 : 0.1

50 : 0.1

All of those combinations end with the following error:

Thu Feb 06 14:20:37 EST 2014 INFO Loading X and Y to test whether they have sufficient rank
Thu Feb 06 14:20:50 EST 2014 INFO Matrix is not yet proved to be non-singular, continuing to load...
Thu Feb 06 14:20:50 EST 2014 WARNING X or Y does not have sufficient rank; deleting this model and its results
Thu Feb 06 14:20:50 EST 2014 INFO Deleting recursively: hdfs://nameservice1/Oryx/data/00000/X
Thu Feb 06 14:20:50 EST 2014 INFO Deleting recursively: hdfs://nameservice1/Oryx/data/00000/Y
Thu Feb 06 14:20:50 EST 2014 INFO Signaling completion of generation 0
Thu Feb 06 14:20:50 EST 2014 INFO Deleting recursively: hdfs://nameservice1/Oryx/data/00000/tmp
Thu Feb 06 14:20:50 EST 2014 INFO Dumping some stats on generation 0
Thu Feb 06 14:20:50 EST 2014 INFO Generation 0 complete

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Thu, 06 Feb 2014 20:32:44 GMT

Hmm. How much data are we talking about? you're building the model on the same data you optimized from?

How many unique users and items?

The general remedy is fewer features and lower lambda, but it can't be right that the optimizer is fine with this while the model build isn't fine with any of those values. Something is not right here...

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Thu, 06 Feb 2014 21:50:27 GMT

The model is indeed being built from the full dataset, while the optimization was performed against a 50% sample. To get the sample, I downloaded the dataset from HDFS to the local filesystem, and performed a "head -n 1883475 data50percent.csv". Then I ran the optimizer locally, not distributed. Should I use the full dataset instead?

Dataset size is 125MB

Number of records 3,766,950

Unique users 608146

Unique items 1151

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Thu, 06 Feb 2014 23:32:54 GMT

I would use 100% of the data, yes, but I don't think that should make a big difference

The number of items is low, but not that low. Is there any reason to think the items are very 'redundant'?

This is strange enough that I think there may be a bug somewhere. Is this data you can share, in anonymized form, offline?

I am wondering whether the singularity threshold needs to be configurable.

I would still try turning down lambda / features to get it going, although I still am not seeing a reason why it should be necessary.

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Fri, 07 Feb 2014 13:08:11 GMT

Some of the items are exceptionally popular, while a large number of the other items have very low values. The weight is a simple count of the items per user within a timeframe. So a userID/itemID combo should only be seen once, but some of those items are seen for a very large percentage of the userIDs.

I've tried setting Features to 5, and Lambda to .01, which also failed. I'll try setting Features to 3 and Lambda to .0001 and see if that has any effect.

I'll verify with our Legal dept about sending the data over, but it shouldn't be an issue. I know I have your card from when we met in London and NY Strata, but it's in my desk at work, and I'm working from home, so you might have to message me with your email address or data drop location.

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Fri, 07 Feb 2014 19:18:45 GMT

I'm cleared to send the dataset, just need to know where it's going!

James

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Fri, 07 Feb 2014 20:09:30 GMT

Thanks James, I'm sowen at cloudera. It sounds like something I will have to debug as it's either quite subtle or a bug in the program. I'll solve it this weekend.

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Sat, 08 Feb 2014 15:57:08 GMT

Thanks James, I got the data.

FWIW, it built successfully locally, which is at least good. That is not a solution, but might get you moving.

I ran the data on Hadoop (CDH5) and uncovered a different problem, which I fixed: https://github.com/cloudera/oryx/commit/fee977f6a682ba6a2e8c2e48275cb4dc5718c8b2

It is basically an artifact of having virtually all even IDs. That shouldn't be a problem of course.

It ran successfully after that. Is the data you showed me the same as what you're running, or a sample (or redacted)? Just want to rationalize why you didn't see the error I did and then see if that has any clue to why it works for me.

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Sat, 08 Feb 2014 19:14:16 GMT

I'll try the local build on one of the datanodes, that shouldn't be a problem for what I'm testing.

It's the full dataset, but the original data was actually userID/prodDesc/weight ... I was informed by our security team that I could send the data if I changed the prodDesc to prodID, since it's pretty meaningless without lookup tables. So the Item variable went from a string when I was testing it, to a numeric; perhaps that's why I didn't see the same error.

So I'm wondering if the problem is only seen if the Item variable is a string ... easy way to test it would be to hash the prodID, which would give an alpha numeric string, similar in format to the original prodDesc.

I can hash the data and re-upload it, or you can run this little bit of python:

#!/usr/bin/python

import csv,hashlib,sys,os,string

INPUT_FILE = csv.reader(open("cloudera_data.csv","rb"), delimiter=",")
OUTPUT_FILE = csv.writer(open("output.csv","wb"), delimiter=",")

for data_lines in INPUT_FILE:
    data_lines[1] = string.upper(hashlib.sha1(string.strip(str(data_lines[1]),chars="\n")).hexdigest())
    OUTPUT_FILE.writerow(data_lines)

sys.exit(0)

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Sat, 08 Feb 2014 21:18:26 GMT

Yes that explains why you didn't see the same initial problem. Well, good that was fixed anyhow.

Text vs numeric shouldn't matter at all. Underneath they are both hashed. Looks the amount of data and its nature are the same if it's just that IDs were hashed. I can't imagine collisions are an issue.

I tried converting these 1-1 to an ID that is alphanumeric, and it worked for me.

You are using CDH 4.x vs 5 right? could be a different, but still don't quite expect a problem would be of this form.

Anything else of interest in the logs? you're welcome to send me all of it.

You're starting from scratch when you run the test ?

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Sat, 08 Feb 2014 22:59:14 GMT

I'm using CDH5 Beta 1, with Oryx compiled against the hadoop22 profile. Speaking of which, you may want to update the Build documentation on github, which states to use profile name "cdh5", but the pom.xml actually uses hadoop22 as the profile name.

I'll try running the test again tonight and see how it works out. If I see anything else, I'll send you the log output, but I'm hoping for the best!

And yes, every test is started from scratch, just in case!

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Sat, 08 Feb 2014 23:08:23 GMT

Oops, fixed. Yes I'm using CDH5b1 too, so that's not a difference. Can you compile from HEAD to make sure we're synced up there? you may already be, just checking. I can make a binary too. Any logs would be of interest for sure. I suppose I would suggest trying again with clearly small values for features (like 10) and clearly small values for lambda (like 0.0001) to see if that at least works. I would expect a lower number of features might be appropriate given there are a smallish number of items. You might try the optimizer again with lower ranges for both. More features encourages overfitting and more lambda encourages underfitting, so they kind of counter-act. It's possible you find a better value when both are low.

Re: Oryx ALS: X and Y do not have sufficient rank

bearrito — Mon, 10 Feb 2014 03:39:05 GMT

I'm able to replicate this issue as well.

I've run through various combinations of lamba/feature pairs. No luck.

I'm running the latest CDH4 binaries.

Sean, would you like my data set?

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Mon, 10 Feb 2014 11:29:54 GMT

It's "normal" for this result to happen if the parameters are way out of kilter for the data set. I suppose it tends to be easier for that to happen with small data. So whether it's reproducing a problem depends on the data. But if you think the params are quite reasonable for the data and you see this, yes please send it to me.

Re: Oryx ALS: X and Y do not have sufficient rank

JamesConner — Mon, 10 Feb 2014 16:01:05 GMT

Hi Sean,

Good news: I recompiled and gave it a whirl giving it 10 Features and .0001 Lambda as a first pass. Nothing abnormal or unusual jumps out at me in the output, so I believe the commits you made did the trick.

Generation 0 was successfully built, and it has passed the X/Y sufficient rank test. At first glance, the recommendations seem valid, if slightly skewed for the most popular items (which is expected). I obviously need to work on a rescorer to minimize the over-represented items. Next thing on the list is to see if it can be fit better, so I obviously need to come up with an automated unit test.

Have you built the Optimizer into the Oryx source code, by chance, or is it just in Myrrix?

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Mon, 10 Feb 2014 16:50:47 GMT

That's good, although I am still not sure why it worked fine for me with quite different params. The transformation should not have done much. It could be that the singularity tolerance is too strict, but I doubt it. There's going to be a fairly big rewrite of the computation, to use Spark in some parts for example. As part of that I am going to build in evaluation to the pipeline itself, so that it's always tuning as it goes. It's not going to come out soon -- just in design phase -- but the idea is that this should not be something anyone has to do by hand. For practical purposes, I would just proceed with these params for now and return to the idea of optimization later. I am guessing (?) your real data set is different anyway and would require different params. Or for this data set you could use the local build.

Re: Oryx ALS: X and Y do not have sufficient rank

srowen — Mon, 08 Dec 2014 00:08:03 GMT

One late reply here: this bug fix may be relevant to the original problem:

https://github.com/cloudera/oryx/issues/99

I'll put this out soon in 1.0.1