- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How can we isolate the jobs running in the same queue?
- Labels:
-
Apache YARN
Created ‎02-01-2016 02:20 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I created four node labels, assigned 4 nodes to each node label. Now, the jobs are running fine in the specified queues and on assigned nodes through node labels. Let us consider a case, two jobs are running in the same queue. Does one job which is acquiring more cluster resources will kill another job in the same queue? How can we isolate two jobs without impacting each other? Is there any solution?
Created ‎02-01-2016 02:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
take a look at job preemption, new feature in YARN. @Ram D you can also raise and lower the priority of each job.
Command to interact with Map Reduce Jobs.
Usage: hadoop job [GENERIC_OPTIONS] [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobOutputDir>] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>]
COMMAND_OPTION | Description |
---|---|
-submit job-file | Submits the job. |
-status job-id | Prints the map and reduce completion percentage and all job counters. |
-counter job-id group-name counter-name | Prints the counter value. |
-kill job-id | Kills the job. |
-events job-id from-event-# #-of-events | Prints the events' details received by jobtracker for the given range. |
-history [all]jobOutputDir | Prints job details, failed and killed tip details. More details about the job such as successful tasks and task attempts made for each task can be viewed by specifying the [all] option. |
-list [all] | Displays jobs which are yet to complete. -list all displays all jobs. |
-kill-task task-id | Kills the task. Killed tasks are NOT counted against failed attempts. |
-fail-task task-id | Fails the task. Failed tasks are counted against failed attempts. |
-set-priority job-id priority | Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW |
Created ‎02-01-2016 02:23 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
take a look at job preemption, new feature in YARN. @Ram D you can also raise and lower the priority of each job.
Command to interact with Map Reduce Jobs.
Usage: hadoop job [GENERIC_OPTIONS] [-submit <job-file>] | [-status <job-id>] | [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobOutputDir>] | [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | [-set-priority <job-id> <priority>]
COMMAND_OPTION | Description |
---|---|
-submit job-file | Submits the job. |
-status job-id | Prints the map and reduce completion percentage and all job counters. |
-counter job-id group-name counter-name | Prints the counter value. |
-kill job-id | Kills the job. |
-events job-id from-event-# #-of-events | Prints the events' details received by jobtracker for the given range. |
-history [all]jobOutputDir | Prints job details, failed and killed tip details. More details about the job such as successful tasks and task attempts made for each task can be viewed by specifying the [all] option. |
-list [all] | Displays jobs which are yet to complete. -list all displays all jobs. |
-kill-task task-id | Kills the task. Killed tasks are NOT counted against failed attempts. |
-fail-task task-id | Fails the task. Failed tasks are counted against failed attempts. |
-set-priority job-id priority | Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW |
Created ‎02-01-2016 02:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How can we isolate two jobs without impacting each other? My requirement is jobs must be run wihtout impacting each other in the same queue. By setting same priority to both jobs is the solution ?
Created ‎02-01-2016 02:38 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Ram D Changes the priority of the job. Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW. Change one job's priority from NORMAL and it will work. You can also look at preemption.
