Support Questions

Find answers, ask questions, and share your expertise

How can we isolate the jobs running in the same queue?

Expert Contributor

I created four node labels, assigned 4 nodes to each node label. Now, the jobs are running fine in the specified queues and on assigned nodes through node labels. Let us consider a case, two jobs are running in the same queue. Does one job which is acquiring more cluster resources will kill another job in the same queue? How can we isolate two jobs without impacting each other? Is there any solution?

1 ACCEPTED SOLUTION

Mentor

take a look at job preemption, new feature in YARN. @Ram D you can also raise and lower the priority of each job.

Command to interact with Map Reduce Jobs.

Usage: hadoop job [GENERIC_OPTIONS] [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobOutputDir>] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>]

COMMAND_OPTIONDescription
-submit job-fileSubmits the job.
-status job-idPrints the map and reduce completion percentage and all job counters.
-counter job-id group-name counter-namePrints the counter value.
-kill job-idKills the job.
-events job-id from-event-# #-of-eventsPrints the events' details received by jobtracker for the given range.
-history [all]jobOutputDirPrints job details, failed and killed tip details. More details about the job such as successful tasks and task attempts made for each task can be viewed by specifying the [all] option.
-list [all]Displays jobs which are yet to complete. -list all displays all jobs.
-kill-task task-idKills the task. Killed tasks are NOT counted against failed attempts.
-fail-task task-idFails the task. Failed tasks are counted against failed attempts.
-set-priority job-id priorityChanges the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW

View solution in original post

3 REPLIES 3

Mentor

take a look at job preemption, new feature in YARN. @Ram D you can also raise and lower the priority of each job.

Command to interact with Map Reduce Jobs.

Usage: hadoop job [GENERIC_OPTIONS] [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobOutputDir>] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>]

COMMAND_OPTIONDescription
-submit job-fileSubmits the job.
-status job-idPrints the map and reduce completion percentage and all job counters.
-counter job-id group-name counter-namePrints the counter value.
-kill job-idKills the job.
-events job-id from-event-# #-of-eventsPrints the events' details received by jobtracker for the given range.
-history [all]jobOutputDirPrints job details, failed and killed tip details. More details about the job such as successful tasks and task attempts made for each task can be viewed by specifying the [all] option.
-list [all]Displays jobs which are yet to complete. -list all displays all jobs.
-kill-task task-idKills the task. Killed tasks are NOT counted against failed attempts.
-fail-task task-idFails the task. Failed tasks are counted against failed attempts.
-set-priority job-id priorityChanges the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW

Expert Contributor

How can we isolate two jobs without impacting each other? My requirement is jobs must be run wihtout impacting each other in the same queue. By setting same priority to both jobs is the solution ?

Mentor

@Ram D Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW. Change one job's priority from NORMAL and it will work. You can also look at preemption.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.