Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oryx 2 with cloudera 5.3

Solved Go to solution

Oryx 2 with cloudera 5.3

New Contributor

Hello Everybody
I am trying to running oryx 2 in Cloudera Express 5.3.0, with a unique node but when i execute the SpeedLayer , this return me the next warning

"WARN YarnClient Cluster Scheduler:71 Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory"

I dont know how solve this problem.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Oryx 2 with cloudera 5.3

Master Collaborator

This means that your YARN cluster does not have the amount of resources (CPU, memory) available that the app is asking for. How much is available?

 

The default is pretty modest though; 2 executors, and 1g RAM and 8 cores per executor. Maybe that's too many cores to ask for? I could turn down the default.

 

You can change it in your config file.

 

oryx = {
...
speed = {
streaming = {
...
# Number of executors to start
num-executors = 2

# Cores per executor
executor-cores = 8

# Memory per executor
executor-memory = "1g"

# Heap size for the Speed driver process
driver-memory = "512m"
...
}

}
6 REPLIES 6

Re: Oryx 2 with cloudera 5.3

Master Collaborator

This means that your YARN cluster does not have the amount of resources (CPU, memory) available that the app is asking for. How much is available?

 

The default is pretty modest though; 2 executors, and 1g RAM and 8 cores per executor. Maybe that's too many cores to ask for? I could turn down the default.

 

You can change it in your config file.

 

oryx = {
...
speed = {
streaming = {
...
# Number of executors to start
num-executors = 2

# Cores per executor
executor-cores = 8

# Memory per executor
executor-memory = "1g"

# Heap size for the Speed driver process
driver-memory = "512m"
...
}

}

Re: Oryx 2 with cloudera 5.3

New Contributor

Thank you very much the mistake was I was only  checking these attributes in the Batch Layer equal that example configuration.

Now  I have a question , what are the necessary machine resources to run oryx 2 properly?

Re: Oryx 2 with cloudera 5.3

Master Collaborator

There is no absolute minimum. I would imagine that, for reasoanbly sized problems, you'd want to allocate 1GB of memory to each of these processes -- meaning at least 1 executor for both batch and streaming with 1GB of memory, and at least 1 and probably more cores. The serving layer should probably have 1GB+ of memory too and will use as many cores as the machine has.

 

This is designed for a cluster of machines, I suppose, but nothing prevents you from running everything on one machine.

Re: Oryx 2 with cloudera 5.3

Explorer

Hi Nelson,  I am trying to running oryx 2 in Cloudera , do have any suggestions or tutorial? How to begin? 

Thanks :)

 

Re: Oryx 2 with cloudera 5.3

Master Collaborator
Have a look at this minimal get-started example:
http://oryxproject.github.io/oryx/docs/endusers.html
Highlighted

Re: Oryx 2 with cloudera 5.3

Explorer

Thanks :)