question Oryx log info of ALS in Archives of Support Questions (Read Only)

Oryx log info of ALS

Jason.Chen — Sat, 23 May 2015 16:13:43 GMT

Sean,

Want to know a little more about Oryx logs as below (ALS computation).

In particular, what's the heap number ? Is it implying the MEM used by Oryx computation layer during the model computation time ?

Sometimes, we see the number is not close to the heap initialized to Oryx, but it signals a warning. So, want to confirm what's the heap number shown below.

Thanks.

Jason

Sat May 23 08:57:48 PDT 2015 INFO 5800000 X/tag rows computed (7876MB heap)
Sat May 23 08:57:50 PDT 2015 INFO 5900000 X/tag rows computed (10487MB heap)
Sat May 23 08:57:53 PDT 2015 INFO 6000000 X/tag rows computed (7108MB heap)

Re: Oryx log info of ALS

srowen — Sat, 23 May 2015 17:39:17 GMT

Yes it is just the current heap usage, which is probably not near the max you set. It is normal. What warning do you mean?

Re: Oryx log info of ALS

Jason.Chen — Sun, 24 May 2015 06:36:49 GMT

I set the heap size as 18GB.

During the ALS computation time, the logs indicates the following MEM warning message. Looks it's because of heap size. One thing confusing is that it indicates 19244MB heap used. If the report is correct, it should drop Out-Of-Memory exception (because my heap size is 18GB which is smaller than 19244 MB). I feel this is confusing.

Thanks.

Jason

Sat May 23 15:36:34 PDT 2015 INFO 3800000 X/tag rows computed (19244MB heap)
Sat May 23 15:36:34 PDT 2015 WARNING Memory is low. Increase heap size with -Xmx, decrease new generation size with larger -XX:NewRatio value, and/or use -XX:+UseCompressedOops

Re: Oryx log info of ALS

srowen — Sun, 24 May 2015 11:30:29 GMT

If you're not seeing a problem you can ignore it. The thing I'd watch for is if you are nearly out of memory and are spending a lot of time in GC. If so then more heap or these other settings might help.

Are you sure the heap is just 18gb? I agree this doesnt quite make sense otherwise. The memory estimate is just that but shouldn't ever be more than the heap total

Re: Oryx log info of ALS

Jason.Chen — Sun, 24 May 2015 16:22:36 GMT

Sean,

Thanks for your reply.

(1) Yes, heap size is set to 18G... Here is what I do (for Oryx ALS computation)

java -Xmx18432m -Dconfig.file=/xxx/oryx.conf -jar /xxx/oryx-computation.jar

(2) A side question: in the Oryx configuration file (https://github.com/cloudera/oryx/blob/master/common/src/main/resources/reference.conf), there are several settings for the computation that I can tune/set from the oryx configuration file.

I think I can just put the settings as part of the Java JVM parameters, and it should replace with the default values inside config file. Confirm?

For example (use model.features and model.alpha as examples)

java -Xmx18432m -Dconfig.file=/xxx/oryx.conf -jar -Dmodel.features=50 -Dmodel.alpha=50 /xxx/oryx-computation.jar

Thanks.

Jason

Re: Oryx log info of ALS

srowen — Sun, 24 May 2015 19:54:27 GMT

Yes it uses Typesafe Config (https://github.com/typesafehub/config) so you should be able to set values on the command line too. Hm, maybe I should change that log to also output the current max heap, if only to be more informative and help debug. I'm not sure why you are seeing that.

Re: Oryx log info of ALS

Jason.Chen — Tue, 09 Jun 2015 02:19:19 GMT

Sean,

I continued to dig into the memory usage, but moved focus to oryx-serving layer.

I noticed from oryx serving log. It indicates the loading of several main objects:

(1) Loaded feature vectors from .../Y/0.csv.gz (this is for item matrix)

(2) Loaded known items from .../knownItems/0.csv.gz

(3) Loaded feature vectors from .../X/0.csv.gz (this is for user matrix)

Based on these loaded objects: I am thinking to compute the MEM used by the model when Oryx serving starts..Assume using feature ranking as 50

(1) (# of items) * 50 * 4 bytes (because each feature vector is with 50 floating points (4 bytes in Java))

(2) MEM requires is the "long" (8 bytes) of user-ID and plus (8 bytes)* (# of known item for each user)

(3) (# of users) * 50 * 4 bytes (because each feature vector is with 50 floating points (4 bytes in Java))

The MEM usage is basically (1) + (2) + (3)

Do I miss any important MEM computation for the Oryx-serving to load ?

Thanks

Re: Oryx log info of ALS

srowen — Mon, 25 May 2015 07:07:22 GMT

That's right, though there's probably a little more than this due to other JVM overheads and other much smaller data structures, but yeah that's a good start at an estimate.

Re: Oryx log info of ALS

Jason.Chen — Tue, 09 Jun 2015 02:21:02 GMT

Sean,

Thanks for the confirmation.

Yes, I understand that there could be more, due to JVM stuff, stack, the code/structures, etc.

I noticed that our oryx-serving uses 9GB MEM after starting up, but the data seems to needs about 1.8GB MEM including

Based on this, I do not understand why there is a big difference from 9GB and 1.8GB... Is my estimation wrong? Any thought ?

Jason

Re: Oryx log info of ALS

srowen — Mon, 25 May 2015 17:19:29 GMT

If you just mean the heap has grown to 9GB, that is normal in the sense that it does not mean 9GB of memory is actually in use. If you have an 18GB heap then a major GC has likely not happened since there is no memory pressure. I would expect this to drop significantly after a major GC. To test, you can force a GC on the running process with "jcmd GC.run" in Java 7+.

Re: Oryx log info of ALS

Jason.Chen — Tue, 09 Jun 2015 02:24:28 GMT

Sean,

Yes, we tried GC and it helped to identify the MEM usage.

We also try to investigate the MEM used by Oryx computation and use Hadoop for model computation.

(1) How to compute those more precisely to know the MEM needed ?

(2) We also ran Oryx in Hadoop and it runs very slow.

Good thing about Hadoop is that it can avoid the Out-of-MEM, but we do want to address the slow

computation of Hadoop. So, my question is that if any suggestions to tune Hadoop stuff in Oryx config

(say, mapper-memory-mb, reducer-memory-mb ?).

(3) We heard that Oryx 2.0 is using Spark and has built-in train-validation process. It looks will help to address the issues

I mentioned in (2) ?

Thanks for your time.

Jason

Re: Oryx log info of ALS

srowen — Thu, 28 May 2015 08:48:00 GMT

You are computing locally rather than on Hadoop right? I don't think there's an easy way to compute memory usage as it will vary somewhat with your parallelism as well as data size. I believe it will require one matrix loaded into memory locally, and that will drive most of the memory usage, and you have an estimate of that. That may help, but, I'd also just measure empirically the heap size to know for sure. You can easily watch the JVM's GC activity with a tool like jprofiler in real time, if you really want to see what's happening. There's no point in using Hadoop if you're just going to run on one machine. It will be an order of magnitude slower as there is a bunch of pointless writes to disk and all the overhead of a full distributed file system and resource scheduler. Hadoop makes sense only if you have a large cluster already, or you need fault tolerance. It sounds like you should simply get a decent estimate of your heap size requirements, which don't sound that large. It sounds like it's well under 9GB? you can easily get a machine in the cloud with tens of GB of RAM. Just do that. Oryx 2 is a completely different architecture. There is no local mode; it's all Hadoop (and Spark). It has a lot of pluses and minuses as a result. I think it would be even worse if you're trying to run on one small machine; it's really for a small cluster at least.

Re: Oryx log info of ALS

Jason.Chen — Tue, 09 Jun 2015 02:26:33 GMT

Sean,

We are experimenting to use (a) single Computation node and (b) single Computation node plus a Hadoop cluster.

We want to see the performance difference in terms of running time for (a) and (b)

Questions:

(1) What do you mean "There's no point in using Hadoop if you're just going to run on one machine." ? Our data will grow up fast and

then we can not just use one VM (and continuously increasing memory). We think Hadoop MapReduce can help us to scale up when data

grows.

(2) Is tuning "mapper-memory-mb" and "reducer-memory-mb" potentially the way to "speed up" the process, as it allocates more MEM ?

Thanks.

Jason

Re: Oryx log info of ALS

srowen — Fri, 29 May 2015 07:50:44 GMT

Yes, that's a good reason, if you have to scale up past one machine. Previously I thought you mean you were running an entire Hadoop cluster on one machine, which is fine for a test but much slower and more complex than a simple non-Hadoop 1-machine setup. I The mapper and reducer will need more memory if you see them running out of memory. If memory is very low but not exhausted, a Java process slows down in too much GC. Otherwise more memory does not help. More nodes does not necessarily help. You still face the overhead of task scheduling and data transfer, and the time taken to do non-distributed work. In fact, if you set up your workers to not live on the same nodes as data nodes, it will be a lot slower. For your scale, which fits in one machine easily, 7 nodes is big overkill, and 60 is way too big to provide any advantage. You're measuring pure Hadoop overhead, which you can tune, but is not reflecting work done. The upshot is you should be able to handle data sizes hundreds or thousands of times larger this way, at roughly the same amount of time. For small data sets, you see why there is no value in trying to use a large cluster; it's just too tiny to split up.