Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flexibility with YARN

Flexibility with YARN

Explorer

Hi Experts,

 

Is it possible to configure YARN on a data node to use half of cpu cores and half of total memory, and when necessary e.g. we are very busy, modify YARN configuration to expand to all available resources (cpu cores, memory)? And when not that busy, shrink back to half of the resouces? 

 

Can YARN be that elastic? Thank you!

 

1 REPLY 1
Highlighted

Re: Flexibility with YARN

Master Guru
One approach would be to have API-performed rolling restarts with config change-sets applied at each schedule. If latency (small drop in available nodes sustained for the duration of batched restarts) isn't an issue, this could work.

If locality isn't important, then you can also achieve this by marking a set of nodes as entirely unavailable via the NodeManager state support, leaving only the rest at full capacity.

This sounds like a scenario best served by Cloud (workload-driven cluster runtimes), offered by Cloudera Altus (or upcoming CDP) and/or Director.

Some links that may be helpful:
- API in CM to perform rolling restart of a specific service: https://archive.cloudera.com/cm6/6.2.0/generic/jar/cm_api/apidocs/resource_ServicesResource.html#res...
- API in CM to apply config changes to a service role group: https://archive.cloudera.com/cm6/6.2.0/generic/jar/cm_api/apidocs/resource_RoleConfigGroupsResource....
- Cloudera Altus: https://www.cloudera.com/documentation/altus/topics/alt_intr_overview.html
Don't have an account?
Coming from Hortonworks? Activate your account here