|Will there be an option to configure resource allocation based on time-periods? for example, i have 2 queues: 1. marketing and 2. etl. from 10 pm to 7:59 am, they want the etl queue to get 70% and marketing 30%. but then from 8am to 9:59pm, etl only 30% and marketing 70%.|
Currently I don't believe there is any such thing. However, you can satisfy these requirements by allocating ETL jobs in sub queue which require different amount of resources along with SLAs. Others you can put in another queue with different min capacity. Same thing for marketing queue. Marketing que will have x% of resources based on any SLAs. Enable preemption as you see fit to meet the SLAs.
Your question is valid and the feature has merit. However my message to you is that capacity scheduler is a true enterprise scheduler which guarantees resources to meet SLA. if you start the discussion with this you will see time based scheduling is not truly the request. It may be masking the inablity to tie SLA to projects or lack of planning & testing.
Assuming that most ETL jobs run over night and only few ETL jobs run during the day, you might "get away" with just configuring elasticity, i.e., configuring:
If you have ETL jobs also started during the day and if you have full control over the queue in which a job is started, you could use the following hack:
You can find a good summary over the advanced options of YARN schedulers here: http://www.slideshare.net/Hadoop_Summit/towards-slabased-scheduling-on-yarn-clusters