question Oryx 2 with cloudera 5.3 in Support Questions

Oryx 2 with cloudera 5.3

Nelson M — Mon, 02 Mar 2015 12:31:38 GMT

Hello Everybody
I am trying to running oryx 2 in Cloudera Express 5.3.0, with a unique node but when i execute the SpeedLayer , this return me the next warning

"WARN YarnClient Cluster Scheduler:71 Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory"

I dont know how solve this problem.

Re: Oryx 2 with cloudera 5.3

srowen — Mon, 02 Mar 2015 12:48:55 GMT

This means that your YARN cluster does not have the amount of resources (CPU, memory) available that the app is asking for. How much is available?

The default is pretty modest though; 2 executors, and 1g RAM and 8 cores per executor. Maybe that's too many cores to ask for? I could turn down the default.

You can change it in your config file.

oryx = {
...
speed = {
  streaming = {
    ...
    # Number of executors to start
    num-executors = 2

    # Cores per executor
    executor-cores = 8

    # Memory per executor
    executor-memory = "1g"

    # Heap size for the Speed driver process
    driver-memory = "512m"
    ...
  }

}

Re: Oryx 2 with cloudera 5.3

Nelson M — Wed, 04 Mar 2015 14:04:16 GMT

Thank you very much the mistake was I was only checking these attributes in the Batch Layer equal that example configuration.

Now I have a question , what are the necessary machine resources to run oryx 2 properly?

Re: Oryx 2 with cloudera 5.3

srowen — Wed, 04 Mar 2015 23:43:49 GMT

There is no absolute minimum. I would imagine that, for reasoanbly sized problems, you'd want to allocate 1GB of memory to each of these processes -- meaning at least 1 executor for both batch and streaming with 1GB of memory, and at least 1 and probably more cores. The serving layer should probably have 1GB+ of memory too and will use as many cores as the machine has.

This is designed for a cluster of machines, I suppose, but nothing prevents you from running everything on one machine.

Re: Oryx 2 with cloudera 5.3

Gustave — Wed, 13 Jan 2016 10:52:47 GMT

Hi Nelson, I am trying to running oryx 2 in Cloudera , do have any suggestions or tutorial? How to begin?

Thanks 🙂

Re: Oryx 2 with cloudera 5.3

srowen — Wed, 13 Jan 2016 11:49:58 GMT

Have a look at this minimal get-started example:
http://oryxproject.github.io/oryx/docs/endusers.html

Re: Oryx 2 with cloudera 5.3

Gustave — Wed, 13 Jan 2016 11:51:39 GMT

Thanks 🙂