Support Questions
Find answers, ask questions, and share your expertise

Distcp failed with Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException

Highlighted

Distcp failed with Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException

Explorer

Hi all,

 

I am using cdh5.1 and just adjusted some parameters for containers/mappers/reducers and now all of my distcp jobs failed for no apparent reasons.  I did some researches and suspected that my yarn.application.classpath so I specified them directly.

 

<!--
<property>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
-->
<property>
<name>yarn.application.classpath</name>
<value>
/etc/hadoop/conf,
/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,
/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,
/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*,
/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*
</value>
</property>

 

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<description> 5632 x 18 containers or 100 Gigs </description>
<value>102400</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>5632</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<description> 5632 x 18 containers or 100 Gigs </description>
<value>102400</value>
</property>

 

---

 

At one point I suspected that this could be caused by the non working job history server but I checked and my job history server is working just fine.

 

My current setup is HA RM but with manual failover...

 

Below are some information on the issues.

 

Thank you very much,

Steven.

 

 

hadoop distcp /user/oozie/profile-v4/output/2014-09-24/success hdfs://active-iad2-hnn/user/oozie/profile-v4/output/2014-09-24/success2
14/09/30 21:51:08 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/user/oozie/profile-v4/output/2014-09-24/success], targetPath=hdfs://active-iad2-hnn/user/oozie/profile-v4/output/2014-09-24/success2, targetPathExists=false}
14/09/30 21:51:09 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
14/09/30 21:51:09 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
14/09/30 21:51:10 INFO mapreduce.JobSubmitter: number of splits:1
14/09/30 21:51:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1412040478812_0516
14/09/30 21:51:10 INFO impl.YarnClientImpl: Submitted application application_1412040478812_0516
14/09/30 21:51:10 INFO mapreduce.Job: The url to track the job: http://sc2-rm1.mynet.com:8100/proxy/application_1412040478812_0516/
14/09/30 21:51:10 INFO tools.DistCp: DistCp job-id: job_1412040478812_0516
14/09/30 21:51:10 INFO mapreduce.Job: Running job: job_1412040478812_0516
14/09/30 21:51:18 INFO mapreduce.Job: Job job_1412040478812_0516 running in uber mode : false
14/09/30 21:51:18 INFO mapreduce.Job: map 0% reduce 0%
14/09/30 21:51:21 INFO mapreduce.Job: Task Id : attempt_1412040478812_0516_m_000000_0, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 1

14/09/30 21:51:25 INFO mapreduce.Job: Task Id : attempt_1412040478812_0516_m_000000_1, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

14/09/30 21:51:31 INFO mapreduce.Job: Task Id : attempt_1412040478812_0516_m_000000_2, Status : FAILED
Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

14/09/30 21:51:36 INFO mapreduce.Job: map 100% reduce 0%
14/09/30 21:51:37 INFO mapreduce.Job: Job job_1412040478812_0516 failed with state FAILED due to: Task failed task_1412040478812_0516_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/09/30 21:51:37 INFO mapreduce.Job: Counters: 11
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=8590
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=8590
Total vcore-seconds taken by all map tasks=8590
Total megabyte-seconds taken by all map tasks=8796160
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
14/09/30 21:51:37 ERROR tools.DistCp: Exception encountered
java.io.IOException: DistCp failure: Job job_1412040478812_0516 has failed: Task failed task_1412040478812_0516_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

at org.apache.hadoop.tools.DistCp.execute(DistCp.java:169)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)

 

 

-------

This is the log in hdfs:

2014-09-30 21:51:16,477 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2014-09-30 21:51:16,495 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 58693
2014-09-30 21:51:16,495 INFO [main] org.mortbay.log: jetty-6.1.26
2014-09-30 21:51:16,530 INFO [main] org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.3.0-cdh5.1.0.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_58693_mapreduce____3i8zgs/webapp
2014-09-30 21:51:16,890 INFO [main] org.mortbay.log: Started SelectChannelConnector@0.0.0.0:58693
2014-09-30 21:51:16,890 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 58693
2014-09-30 21:51:17,399 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2014-09-30 21:51:17,406 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2014-09-30 21:51:17,407 INFO [Socket Reader #1 for port 19550] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 19550
2014-09-30 21:51:17,413 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-09-30 21:51:17,414 INFO [IPC Server listener on 19550] org.apache.hadoop.ipc.Server: IPC Server listener on 19550: starting
2014-09-30 21:51:17,437 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2014-09-30 21:51:17,437 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2014-09-30 21:51:17,437 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2014-09-30 21:51:17,533 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-09-30 21:51:17,543 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2014-09-30 21:51:17,644 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: 8192
2014-09-30 21:51:17,644 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: default
2014-09-30 21:51:17,650 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2014-09-30 21:51:17,655 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500

2014-09-30 21:51:17,672 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1412040478812_0516Job Transitioned from INITED to SETUP
2014-09-30 21:51:17,683 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2014-09-30 21:51:17,697 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1412040478812_0516Job Transitioned from SETUP to RUNNING
2014-09-30 21:51:17,721 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1412040478812_0516_m_000000 Task Transitioned from NEW to SCHEDULED
2014-09-30 21:51:17,723 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2014-09-30 21:51:17,724 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceReqt:1024
2014-09-30 21:51:17,785 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1412040478812_0516, File: hdfs://sc2prod/user/steven/.staging/job_1412040478812_0516/job_1412040478812_0516_1.jhist
2014-09-30 21:51:18,648 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:18,692 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:5311488, vCores:0> knownNMs=90
2014-09-30 21:51:19,706 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2014-09-30 21:51:19,721 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved sc2-hmr105.mynet.com to /SC2/P1/C7
2014-09-30 21:51:19,724 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1412040478812_0516_01_000002 to attempt_1412040478812_0516_m_000000_0
2014-09-30 21:51:19,726 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:19,785 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sc2-hmr105.mynet.com to /SC2/P1/C7

2014-09-30 21:51:19,805 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://sc2prod/user/steven/.staging/job_1412040478812_0516/job.jar
2014-09-30 21:51:19,808 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/steven/.staging/job_1412040478812_0516/job.xml
2014-09-30 21:51:19,809 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
2014-09-30 21:51:19,809 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
2014-09-30 21:51:19,809 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2014-09-30 21:51:19,838 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2014-09-30 21:51:19,843 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1412040478812_0516_01_000002 taskAttempt attempt_1412040478812_0516_m_000000_0
2014-09-30 21:51:19,846 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1412040478812_0516_m_000000_0
2014-09-30 21:51:19,850 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sc2-hmr105.mynet.com:29331
2014-09-30 21:51:19,925 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1412040478812_0516_m_000000_0 : 13562
2014-09-30 21:51:19,927 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1412040478812_0516_m_000000_0] using containerId: [container_1412040478812_0516_01_000002 on NM: [sc2-hmr105.mynet.com:29331]
2014-09-30 21:51:19,933 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING

2014-09-30 21:51:20,728 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:5336064, vCores:-1> knownNMs=90
2014-09-30 21:51:21,735 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1412040478812_0516_01_000002
2014-09-30 21:51:21,736 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:21,741 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2014-09-30 21:51:21,741 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1412040478812_0516_m_000000_0: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

2014-09-30 21:51:21,743 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1412040478812_0516_m_000000_0
2014-09-30 21:51:21,768 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2014-09-30 21:51:21,769 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2014-09-30 21:51:21,782 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://sc2prod/user/steven/.staging/_distcp-1539285956/_logs/_temporary/1/_temporary/attempt_1412040478812_0516_m_000000_0
2014-09-30 21:51:21,784 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2014-09-30 21:51:21,793 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sc2-hmr105.mynet.com
2014-09-30 21:51:21,831 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2014-09-30 21:51:21,832 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1412040478812_0516_m_000000_1 to list of failed maps
2014-09-30 21:51:22,736 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:22,738 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:5336064, vCores:-1> knownNMs=90
2014-09-30 21:51:23,740 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2014-09-30 21:51:23,740 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1412040478812_0516_01_000003, NodeId: sc2-hmr105.mynet.com:29331, NodeHttpAddress: sc2-hmr105.mynet.com:23999, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.80.1.187:29331 }, ] to fast fail map
2014-09-30 21:51:23,740 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
2014-09-30 21:51:23,741 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1412040478812_0516_01_000003 to attempt_1412040478812_0516_m_000000_1

2014-09-30 21:51:23,741 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:23,741 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sc2-hmr105.mynet.com to /SC2/P1/C7
2014-09-30 21:51:23,742 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2014-09-30 21:51:23,743 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1412040478812_0516_01_000003 taskAttempt attempt_1412040478812_0516_m_000000_1
2014-09-30 21:51:23,744 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1412040478812_0516_m_000000_1
2014-09-30 21:51:23,748 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1412040478812_0516_m_000000_1 : 13562
2014-09-30 21:51:23,748 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1412040478812_0516_m_000000_1] using containerId: [container_1412040478812_0516_01_000003 on NM: [sc2-hmr105.mynet.com:29331]
2014-09-30 21:51:23,749 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
2014-09-30 21:51:24,743 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:5348352, vCores:-1> knownNMs=90
2014-09-30 21:51:25,744 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1412040478812_0516_01_000003
2014-09-30 21:51:25,745 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:25,745 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_1 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2014-09-30 21:51:25,745 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1412040478812_0516_m_000000_1: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:

org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

2014-09-30 21:51:25,747 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1412040478812_0516_01_000003 taskAttempt attempt_1412040478812_0516_m_000000_1
2014-09-30 21:51:25,747 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1412040478812_0516_m_000000_1
2014-09-30 21:51:25,750 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_1 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2014-09-30 21:51:25,751 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2014-09-30 21:51:25,753 WARN [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://sc2prod/user/steven/.staging/_distcp-1539285956/_logs/_temporary/1/_temporary/attempt_1412040478812_0516_m_000000_1
2014-09-30 21:51:25,754 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_1 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2014-09-30 21:51:25,754 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node sc2-hmr105.mynet.com

2014-09-30 21:51:25,755 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1412040478812_0516_m_000000_2 to list of failed maps
2014-09-30 21:51:26,745 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:26,751 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:4653056, vCores:0> knownNMs=78
2014-09-30 21:51:27,759 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2014-09-30 21:51:27,759 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1412040478812_0516_01_000004, NodeId: sc2-hmr100.mynet.com:36775, NodeHttpAddress: sc2-hmr100.mynet.com:23999, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.80.1.182:36775 }, ] to fast fail map
2014-09-30 21:51:27,759 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
2014-09-30 21:51:27,760 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1412040478812_0516_01_000004 to attempt_1412040478812_0516_m_000000_2
2014-09-30 21:51:27,760 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:27,768 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sc2-hmr100.mynet.com to /SC2/P1/C8
2014-09-30 21:51:27,769 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2014-09-30 21:51:27,770 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1412040478812_0516_01_000004 taskAttempt attempt_1412040478812_0516_m_000000_2
2014-09-30 21:51:27,770 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1412040478812_0516_m_000000_2
2014-09-30 21:51:27,770 INFO [ContainerLauncher #4] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sc2-hmr100.mynet.com:36775
2014-09-30 21:51:28,067 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1412040478812_0516_m_000000_2 : 13562

2014-09-30 21:51:28,067 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1412040478812_0516_m_000000_2] using containerId: [container_1412040478812_0516_01_000004 on NM: [sc2-hmr100.mynet.com:36775]
2014-09-30 21:51:28,068 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
2014-09-30 21:51:28,762 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:4556800, vCores:-1> knownNMs=78
2014-09-30 21:51:30,766 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1412040478812_0516_01_000004
2014-09-30 21:51:30,766 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:30,766 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_2 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2014-09-30 21:51:30,767 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1412040478812_0516_m_000000_2: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

2014-09-30 21:51:30,768 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1412040478812_0516_m_000000_2
2014-09-30 21:51:30,783 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_2 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2014-09-30 21:51:30,784 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2014-09-30 21:51:30,786 WARN [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://sc2prod/user/steven/.staging/_distcp-1539285956/_logs/_temporary/1/_temporary/attempt_1412040478812_0516_m_000000_2
2014-09-30 21:51:30,787 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_2 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2014-09-30 21:51:30,788 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sc2-hmr100.mynet.com
2014-09-30 21:51:30,788 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_3 TaskAttempt Transitioned from NEW to UNASSIGNED
2014-09-30 21:51:30,788 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1412040478812_0516_m_000000_3 to list of failed maps
2014-09-30 21:51:31,766 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:31,768 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:4644864, vCores:-1> knownNMs=79
2014-09-30 21:51:32,770 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2014-09-30 21:51:32,770 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_1412040478812_0516_01_000005, NodeId: sc2-hmr105.mynet.com:29331, NodeHttpAddress: sc2-hmr105.mynet.com:23999, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.80.1.187:29331 }, ] to fast fail map
2014-09-30 21:51:32,770 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
2014-09-30 21:51:32,771 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1412040478812_0516_01_000005 to attempt_1412040478812_0516_m_000000_3

2014-09-30 21:51:32,771 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sc2-hmr105.mynet.com to /SC2/P1/C7
2014-09-30 21:51:32,772 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2014-09-30 21:51:32,773 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1412040478812_0516_01_000005 taskAttempt attempt_1412040478812_0516_m_000000_3
2014-09-30 21:51:32,773 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1412040478812_0516_m_000000_3
2014-09-30 21:51:32,778 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1412040478812_0516_m_000000_3 : 13562
2014-09-30 21:51:32,778 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1412040478812_0516_m_000000_3] using containerId: [container_1412040478812_0516_01_000005 on NM: [sc2-hmr105.mynet.com:29331]
2014-09-30 21:51:32,779 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_3 TaskAttempt Transitioned from ASSIGNED to RUNNING
2014-09-30 21:51:33,776 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1412040478812_0516: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:4710400, vCores:-1> knownNMs=79
2014-09-30 21:51:34,778 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1412040478812_0516_01_000005
2014-09-30 21:51:34,778 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:0 RackLocal:0
2014-09-30 21:51:34,778 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_3 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2014-09-30 21:51:34,778 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1412040478812_0516_m_000000_3: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Container exited with a non-zero exit code 1

2014-09-30 21:51:34,780 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1412040478812_0516_01_000005 taskAttempt attempt_1412040478812_0516_m_000000_3
2014-09-30 21:51:34,780 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1412040478812_0516_m_000000_3
2014-09-30 21:51:34,783 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_3 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2014-09-30 21:51:34,784 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2014-09-30 21:51:34,786 WARN [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://sc2prod/user/steven/.staging/_distcp-1539285956/_logs/_temporary/1/_temporary/attempt_1412040478812_0516_m_000000_3
2014-09-30 21:51:34,786 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1412040478812_0516_m_000000_3 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2014-09-30 21:51:34,789 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1412040478812_0516_m_000000 Task Transitioned from RUNNING to FAILED
2014-09-30 21:51:34,789 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node sc2-hmr105.mynet.com
2014-09-30 21:51:34,789 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host sc2-hmr105.mynet.com
2014-09-30 21:51:34,789 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2014-09-30 21:51:34,790 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0

2014-09-30 21:51:34,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1412040478812_0516Job Transitioned from RUNNING to FAIL_ABORT
2014-09-30 21:51:34,791 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
2014-09-30 21:51:35,338 INFO [CommitterEvent Processor #0] org.apache.hadoop.tools.mapred.CopyCommitter: Cleaning up temporary work folder: /user/steven/.staging/_distcp-1539285956
2014-09-30 21:51:35,346 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1412040478812_0516Job Transitioned from FAIL_ABORT to FAILED
2014-09-30 21:51:35,347 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2014-09-30 21:51:35,347 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2014-09-30 21:51:35,347 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2014-09-30 21:51:35,347 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2014-09-30 21:51:35,347 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2014-09-30 21:51:35,347 INFO [Thread-68] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2014-09-30 21:51:35,348 INFO [Thread-68] org.apache.hadoop.mapreduce.jobhistory.JobHisto

Don't have an account?