Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to set max number jobs for a special datanode

How to set max number jobs for a special datanode

New Contributor

Hi

One node in my cluster(on yarn) has some other applications installed, so if i can limit the max job numbers in this node will be good for these applications run quickly.

And is this possible? 

 

 

Very Thanks!

1 REPLY 1
Highlighted

Re: How to set max number jobs for a special datanode

Champion
Do you mean you have apps installed outside of Hadoop and that they are not install on all only on certain nodes?

First, I do not recommend that. The worker nodes do the work and anything that detracts from that decreases the value they bring to the cluster.

In the event that you need an app or binary so be used by jobs, you should have it on all workers. Then you can keep the configs the same across all nodes.

The title and description make me a bit confused. So I am assuming that you have some apps installed on a worker nodes that are not part of Hadoop or CDH. If that is correct, the best way to handle it is to have two separate host templates for YARN that will give less resources to YARN on the host(s) that have less resources to give due to the external applications.

On the topic of limiting jobs, you might be able to use Node labels. These are used to ensure certain types of jobs run on specific nodes. So you could use them to control what runs on that box, but it isn't as cut and dry as I think you expect.
Don't have an account?
Coming from Hortonworks? Activate your account here