Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Oryx 2 with cloudera 5.3

avatar
New Contributor

Hello Everybody
I am trying to running oryx 2 in Cloudera Express 5.3.0, with a unique node but when i execute the SpeedLayer , this return me the next warning

"WARN YarnClient Cluster Scheduler:71 Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory"

I dont know how solve this problem.

1 ACCEPTED SOLUTION

avatar
Master Collaborator

This means that your YARN cluster does not have the amount of resources (CPU, memory) available that the app is asking for. How much is available?

 

The default is pretty modest though; 2 executors, and 1g RAM and 8 cores per executor. Maybe that's too many cores to ask for? I could turn down the default.

 

You can change it in your config file.

 

oryx = {
...
speed = {
streaming = {
...
# Number of executors to start
num-executors = 2

# Cores per executor
executor-cores = 8

# Memory per executor
executor-memory = "1g"

# Heap size for the Speed driver process
driver-memory = "512m"
...
}

}

View solution in original post

6 REPLIES 6

avatar
Master Collaborator

This means that your YARN cluster does not have the amount of resources (CPU, memory) available that the app is asking for. How much is available?

 

The default is pretty modest though; 2 executors, and 1g RAM and 8 cores per executor. Maybe that's too many cores to ask for? I could turn down the default.

 

You can change it in your config file.

 

oryx = {
...
speed = {
streaming = {
...
# Number of executors to start
num-executors = 2

# Cores per executor
executor-cores = 8

# Memory per executor
executor-memory = "1g"

# Heap size for the Speed driver process
driver-memory = "512m"
...
}

}

avatar
New Contributor

Thank you very much the mistake was I was only  checking these attributes in the Batch Layer equal that example configuration.

Now  I have a question , what are the necessary machine resources to run oryx 2 properly?

avatar
Master Collaborator

There is no absolute minimum. I would imagine that, for reasoanbly sized problems, you'd want to allocate 1GB of memory to each of these processes -- meaning at least 1 executor for both batch and streaming with 1GB of memory, and at least 1 and probably more cores. The serving layer should probably have 1GB+ of memory too and will use as many cores as the machine has.

 

This is designed for a cluster of machines, I suppose, but nothing prevents you from running everything on one machine.

avatar
Explorer

Hi Nelson,  I am trying to running oryx 2 in Cloudera , do have any suggestions or tutorial? How to begin? 

Thanks 🙂

 

avatar
Master Collaborator
Have a look at this minimal get-started example:
http://oryxproject.github.io/oryx/docs/endusers.html

avatar
Explorer

Thanks 🙂