Queue configuration in MapReduce

[#1]

Hi,

I have a MapReduce job which is running over more than 170 million records. This is resulting into consuming 98% of queue resource & 89% of cluster resource . Admin team is recommending that they will create new queue with limited configuration, and I should push my job into that queue.

Here are questions, I have :-

1- How can I push my mapreduce job ("hadoop jar") with minimal change to new queue ?

2- As newly created queue has limited resources, what if queue's capacity is full ? Will it result into long run or Job failure ?

3- Is there any other optimal way to prevent job from consuming all resource, we are ok if job runs little longer.

Please advise. Any help will be great.

Thanks

[#2]

Question1: this has already been answered here.

Question 2: Yes it will result in long job run time. The job will be queue'd up until your queue has resource. The admin can create a strategy to allow your queue to take up more resources if they are available on your cluster and enable pre-emption.

Question 3: You can set the amount of ram you mappers and reducers take for your job and see the end outcome. This will be trial and error. There are awesome tools like Dr. Elephant or Unravel Data who focus strictly on job optimizations (APM tools)