Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Capacity Scheduler: Resource allocation based on time-periods

Capacity Scheduler: Resource allocation based on time-periods

Will there be an option to configure resource allocation based on time-periods? for example, i have 2 queues: 1. marketing and 2. etl.   from 10 pm to 7:59 am, they want the etl queue to get 70% and marketing 30%. but then from 8am to 9:59pm, etl only 30% and marketing 70%. 
2 REPLIES 2

Re: Capacity Scheduler: Resource allocation based on time-periods

Super Guru

@Binu Mathew

Currently I don't believe there is any such thing. However, you can satisfy these requirements by allocating ETL jobs in sub queue which require different amount of resources along with SLAs. Others you can put in another queue with different min capacity. Same thing for marketing queue. Marketing que will have x% of resources based on any SLAs. Enable preemption as you see fit to meet the SLAs.

Your question is valid and the feature has merit. However my message to you is that capacity scheduler is a true enterprise scheduler which guarantees resources to meet SLA. if you start the discussion with this you will see time based scheduling is not truly the request. It may be masking the inablity to tie SLA to projects or lack of planning & testing.

Highlighted

Re: Capacity Scheduler: Resource allocation based on time-periods

New Contributor
@Binu Mathew

Assuming that most ETL jobs run over night and only few ETL jobs run during the day, you might "get away" with just configuring elasticity, i.e., configuring:

  • etl queue with capacity 70%, max capacity 70%
  • marketing queue with capacity 30%, max capacity 70%

If you have ETL jobs also started during the day and if you have full control over the queue in which a job is started, you could use the following hack:

  • etl_night queue with capacity 30%, max capacity 70% (only use this queue during night)
  • etl_day queue with capacity 30%, max capacity 30% (only use this queue during day)
  • marketing queue with capacity 30%, max capacity 70% (only use this queue during day)
  • default queue with capacity 10%, max capacity 10%

You can find a good summary over the advanced options of YARN schedulers here: http://www.slideshare.net/Hadoop_Summit/towards-slabased-scheduling-on-yarn-clusters

Unfortunately, there is not exactly what you are asking for. The closest thing to you question might be the (experimental?) reservation system from YARN-1051 and YARN-2572 (wip).

Don't have an account?
Coming from Hortonworks? Activate your account here