Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

wordcount not completed with Yarn exception ...

Highlighted

wordcount not completed with Yarn exception ...

Explorer

I am trying to perform a simple wordcount on my single node server (Ubuntu-Hadoop 2.7.2 - not HDP) and as soon as the map reduce process start I receive the below error in the log...

Sounds this is due to some memory issue but not sure what kind of issue and less sure how to fix it...

Could someone help?

thanks!

******************************

2016-02-16 13:24:00,063 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000005
2016-02-16 13:24:00,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:00,111 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14955 for container-id container_1455610525380_0004_01_000005: 78.8 MB of 1 GB physical memory used; 746.3 MB of 2.1 GB virtual memory used
2016-02-16 13:24:01,933 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000005 is : 1
2016-02-16 13:24:01,933 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000005 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
************************************
11 REPLIES 11
Highlighted

Re: wordcount not completed with Yarn exception ...

@luc tiber Need more info from logs

Highlighted

Re: wordcount not completed with Yarn exception ...

Mentor

check your nodemanager logs, this is a generic error and can't say much. @luc tiber

Highlighted

Re: wordcount not completed with Yarn exception ...

Explorer

@Neeraj Sabharwal

here is the complete yarn log (the nodemanager log is below)

2016-02-16 13:23:22,886 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000001 by user hduser
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1455610525380_0004
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from NEW to INITING
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from INITING to RUNNING
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000001 to application application_1455610525380_0004
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from NEW to LOCALIZING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.jar transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.splitmetainfo transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.split transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.xml transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1455610525380_0004_01_000001
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000001
2016-02-16 13:23:22,896 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1455610525380_0004_01_000001.tokens. Credentials list: 
2016-02-16 13:23:22,913 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hduser
2016-02-16 13:23:22,915 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1455610525380_0004_01_000001.tokens to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000001.tokens
2016-02-16 13:23:22,915 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004 = file:/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004
2016-02-16 13:23:23,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.jar(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/10/job.jar) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,025 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.splitmetainfo(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/11/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,035 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.split(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/12/job.split) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.xml(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:23,067 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:23,070 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000001/default_container_executor.sh]
2016-02-16 13:23:23,755 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000001
2016-02-16 13:23:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 48.4 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-02-16 13:23:26,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 96.5 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-02-16 13:23:29,808 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 115.0 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:32,074 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:32,079 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000002 by user hduser
2016-02-16 13:23:32,079 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000002 to application application_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from NEW to LOCALIZING
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:32,080 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000002
2016-02-16 13:23:32,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:32,113 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000002/default_container_executor.sh]
2016-02-16 13:23:32,808 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000002
2016-02-16 13:23:32,826 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14819 for container-id container_1455610525380_0004_01_000002: 49.2 MB of 1 GB physical memory used; 713.4 MB of 2.1 GB virtual memory used
2016-02-16 13:23:32,846 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 125.9 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:35,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14819 for container-id container_1455610525380_0004_01_000002: 78.3 MB of 1 GB physical memory used; 747.3 MB of 2.1 GB virtual memory used
2016-02-16 13:23:35,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.9 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:37,544 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000002 is : 1
2016-02-16 13:23:37,544 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000002 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000002
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:37,546 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:23:37,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:23:37,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000002
2016-02-16 13:23:37,570 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000002
2016-02-16 13:23:37,571 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000002
2016-02-16 13:23:37,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:23:37,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000002 from application application_1455610525380_0004
2016-02-16 13:23:37,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:23:37,982 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:37,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000002
2016-02-16 13:23:37,986 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000002
2016-02-16 13:23:38,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000002
2016-02-16 13:23:38,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:38,989 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000002]
2016-02-16 13:23:39,989 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:39,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000003 by user hduser
2016-02-16 13:23:39,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000003 to application application_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from NEW to LOCALIZING
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:39,995 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:39,999 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000003
2016-02-16 13:23:40,014 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:40,018 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000003/default_container_executor.sh]
2016-02-16 13:23:41,893 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000003
2016-02-16 13:23:41,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14866 for container-id container_1455610525380_0004_01_000003: 55.7 MB of 1 GB physical memory used; 713.4 MB of 2.1 GB virtual memory used
2016-02-16 13:23:41,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.4 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:44,945 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14866 for container-id container_1455610525380_0004_01_000003: 88.0 MB of 1 GB physical memory used; 651.4 MB of 2.1 GB virtual memory used
2016-02-16 13:23:44,961 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.3 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:44,966 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000003 is : 1
2016-02-16 13:23:44,966 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000003 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000003
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:44,967 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000003
2016-02-16 13:23:44,984 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000003
2016-02-16 13:23:44,985 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000003
2016-02-16 13:23:44,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:23:44,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000003 from application application_1455610525380_0004
2016-02-16 13:23:44,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:23:46,001 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:46,008 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000003
2016-02-16 13:23:46,008 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000003
2016-02-16 13:23:47,010 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000003]
2016-02-16 13:23:47,962 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000003
2016-02-16 13:23:47,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.3 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:48,006 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:48,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000004 by user hduser
2016-02-16 13:23:48,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000004 to application application_1455610525380_0004
2016-02-16 13:23:48,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from NEW to LOCALIZING
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:48,013 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:48,017 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000004
2016-02-16 13:23:48,033 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:48,036 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000004/default_container_executor.sh]
2016-02-16 13:23:50,971 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000004
2016-02-16 13:23:50,990 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.3 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:51,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14909 for container-id container_1455610525380_0004_01_000004: 80.6 MB of 1 GB physical memory used; 746.3 MB of 2.1 GB virtual memory used
2016-02-16 13:23:52,711 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000004 is : 1
2016-02-16 13:23:52,711 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000004 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000004
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:52,712 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:23:52,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:23:52,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000004
2016-02-16 13:23:52,735 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000004
2016-02-16 13:23:52,736 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000004
2016-02-16 13:23:52,736 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:23:52,736 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000004 from application application_1455610525380_0004
2016-02-16 13:23:52,736 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:23:54,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000004
2016-02-16 13:23:54,024 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:54,030 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000004
2016-02-16 13:23:54,030 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000004
2016-02-16 13:23:54,039 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 116.9 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:55,032 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000004]
2016-02-16 13:23:57,031 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:57,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000005 by user hduser
2016-02-16 13:23:57,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000005 to application application_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from NEW to LOCALIZING
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:57,037 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:57,041 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000005
2016-02-16 13:23:57,061 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:57,065 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:57,068 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000005/default_container_executor.sh]
2016-02-16 13:24:00,063 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000005
2016-02-16 13:24:00,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:00,111 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14955 for container-id container_1455610525380_0004_01_000005: 78.8 MB of 1 GB physical memory used; 746.3 MB of 2.1 GB virtual memory used
2016-02-16 13:24:01,933 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000005 is : 1
2016-02-16 13:24:01,933 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000005 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000005
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:24:01,934 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000005
2016-02-16 13:24:01,956 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000005
2016-02-16 13:24:01,957 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000005
2016-02-16 13:24:01,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:24:01,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000005 from application application_1455610525380_0004
2016-02-16 13:24:01,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:24:03,063 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:24:03,069 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000005
2016-02-16 13:24:03,069 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000005
2016-02-16 13:24:03,112 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000005
2016-02-16 13:24:03,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.2 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:06,143 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 118.7 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:09,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 118.7 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:09,273 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_1455610525380_0004_01_000001 succeeded 
2016-02-16 13:24:09,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
2016-02-16 13:24:09,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000001
2016-02-16 13:24:09,290 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000001
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - SucceededTARGET=ContainerImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000001
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000001 from application application_1455610525380_0004
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:24:10,083 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:24:10,086 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000001
2016-02-16 13:24:10,086 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000001
2016-02-16 13:24:10,087 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000001, container_1455610525380_0004_01_000005]
2016-02-16 13:24:10,087 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1455610525380_0004
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1455610525380_0004, with delay of 10800 seconds
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004
2016-02-16 13:24:12,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000001
Highlighted

Re: wordcount not completed with Yarn exception ...

Explorer

@Artem Ervits

My mistake... Thank you for pointing out for me.

2016-02-16 13:23:22,886 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000001 by user hduser
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1455610525380_0004
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from NEW to INITING
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from INITING to RUNNING
2016-02-16 13:23:22,891 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000001 to application application_1455610525380_0004
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from NEW to LOCALIZING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.jar transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.splitmetainfo transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.split transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.xml transitioned from INIT to DOWNLOADING
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1455610525380_0004_01_000001
2016-02-16 13:23:22,892 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000001
2016-02-16 13:23:22,896 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1455610525380_0004_01_000001.tokens. Credentials list: 
2016-02-16 13:23:22,913 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hduser
2016-02-16 13:23:22,915 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /tmp/hadoop-hduser/nm-local-dir/nmPrivate/container_1455610525380_0004_01_000001.tokens to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000001.tokens
2016-02-16 13:23:22,915 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004 = file:/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004
2016-02-16 13:23:23,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.jar(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/10/job.jar) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,025 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.splitmetainfo(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/11/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,035 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.split(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/12/job.split) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://singlenode:9000/tmp/hadoop-yarn/staging/hduser/.staging/job_1455610525380_0004/job.xml(->/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED
2016-02-16 13:23:23,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:23,067 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:23,070 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000001/default_container_executor.sh]
2016-02-16 13:23:23,755 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000001
2016-02-16 13:23:23,777 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 48.4 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-02-16 13:23:26,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 96.5 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-02-16 13:23:29,808 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 115.0 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:32,074 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:32,079 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000002 by user hduser
2016-02-16 13:23:32,079 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000002 to application application_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from NEW to LOCALIZING
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:32,080 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:32,080 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000002
2016-02-16 13:23:32,110 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:32,113 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000002/default_container_executor.sh]
2016-02-16 13:23:32,808 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000002
2016-02-16 13:23:32,826 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14819 for container-id container_1455610525380_0004_01_000002: 49.2 MB of 1 GB physical memory used; 713.4 MB of 2.1 GB virtual memory used
2016-02-16 13:23:32,846 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 125.9 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:35,862 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14819 for container-id container_1455610525380_0004_01_000002: 78.3 MB of 1 GB physical memory used; 747.3 MB of 2.1 GB virtual memory used
2016-02-16 13:23:35,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.9 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:37,544 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000002 is : 1
2016-02-16 13:23:37,544 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000002 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000002
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:23:37,545 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:37,546 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:23:37,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:23:37,546 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000002
2016-02-16 13:23:37,570 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000002
2016-02-16 13:23:37,571 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000002
2016-02-16 13:23:37,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000002 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:23:37,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000002 from application application_1455610525380_0004
2016-02-16 13:23:37,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:23:37,982 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:37,986 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000002
2016-02-16 13:23:37,986 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000002
2016-02-16 13:23:38,884 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000002
2016-02-16 13:23:38,892 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:38,989 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000002]
2016-02-16 13:23:39,989 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:39,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000003 by user hduser
2016-02-16 13:23:39,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000003 to application application_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from NEW to LOCALIZING
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:39,995 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:39,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:39,999 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000003
2016-02-16 13:23:40,014 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:40,018 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000003/default_container_executor.sh]
2016-02-16 13:23:41,893 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000003
2016-02-16 13:23:41,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14866 for container-id container_1455610525380_0004_01_000003: 55.7 MB of 1 GB physical memory used; 713.4 MB of 2.1 GB virtual memory used
2016-02-16 13:23:41,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.4 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:44,945 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14866 for container-id container_1455610525380_0004_01_000003: 88.0 MB of 1 GB physical memory used; 651.4 MB of 2.1 GB virtual memory used
2016-02-16 13:23:44,961 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.3 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:44,966 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000003 is : 1
2016-02-16 13:23:44,966 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000003 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000003
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:23:44,966 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:44,967 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:23:44,967 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000003
2016-02-16 13:23:44,984 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000003
2016-02-16 13:23:44,985 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000003
2016-02-16 13:23:44,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000003 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:23:44,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000003 from application application_1455610525380_0004
2016-02-16 13:23:44,985 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:23:46,001 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:46,008 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000003
2016-02-16 13:23:46,008 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000003
2016-02-16 13:23:47,010 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000003]
2016-02-16 13:23:47,962 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000003
2016-02-16 13:23:47,970 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.3 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:48,006 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:48,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000004 by user hduser
2016-02-16 13:23:48,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000004 to application application_1455610525380_0004
2016-02-16 13:23:48,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from NEW to LOCALIZING
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:48,013 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:48,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:48,017 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000004
2016-02-16 13:23:48,033 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:48,036 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000004/default_container_executor.sh]
2016-02-16 13:23:50,971 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000004
2016-02-16 13:23:50,990 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.3 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:51,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14909 for container-id container_1455610525380_0004_01_000004: 80.6 MB of 1 GB physical memory used; 746.3 MB of 2.1 GB virtual memory used
2016-02-16 13:23:52,711 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000004 is : 1
2016-02-16 13:23:52,711 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000004 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000004
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:23:52,712 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:23:52,712 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:23:52,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:23:52,716 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000004
2016-02-16 13:23:52,735 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000004
2016-02-16 13:23:52,736 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000004
2016-02-16 13:23:52,736 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000004 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:23:52,736 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000004 from application application_1455610525380_0004
2016-02-16 13:23:52,736 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:23:54,012 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000004
2016-02-16 13:23:54,024 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:54,030 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000004
2016-02-16 13:23:54,030 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000004
2016-02-16 13:23:54,039 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 116.9 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:55,032 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000004]
2016-02-16 13:23:57,031 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:23:57,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1455610525380_0004_01_000005 by user hduser
2016-02-16 13:23:57,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1455610525380_0004_01_000005 to application application_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from NEW to LOCALIZING
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle
2016-02-16 13:23:57,037 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1455610525380_0004
2016-02-16 13:23:57,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from LOCALIZING to LOCALIZED
2016-02-16 13:23:57,041 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Start Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000005
2016-02-16 13:23:57,061 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:23:57,065 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from LOCALIZED to RUNNING
2016-02-16 13:23:57,068 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000005/default_container_executor.sh]
2016-02-16 13:24:00,063 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1455610525380_0004_01_000005
2016-02-16 13:24:00,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.1 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:00,111 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14955 for container-id container_1455610525380_0004_01_000005: 78.8 MB of 1 GB physical memory used; 746.3 MB of 2.1 GB virtual memory used
2016-02-16 13:24:01,933 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1455610525380_0004_01_000005 is : 1
2016-02-16 13:24:01,933 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1455610525380_0004_01_000005 and exit code: 1
ExitCodeException exitCode=1: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1455610525380_0004_01_000005
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:456)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
2016-02-16 13:24:01,933 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2016-02-16 13:24:01,934 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from RUNNING to EXITED_WITH_FAILURE
2016-02-16 13:24:01,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000005
2016-02-16 13:24:01,956 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000005
2016-02-16 13:24:01,957 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - FailedTARGET=ContainerImplRESULT=FAILUREDESCRIPTION=Container failed with state: EXITED_WITH_FAILUREAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000005
2016-02-16 13:24:01,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000005 transitioned from EXITED_WITH_FAILURE to DONE
2016-02-16 13:24:01,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000005 from application application_1455610525380_0004
2016-02-16 13:24:01,957 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:24:03,063 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:24:03,069 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000005
2016-02-16 13:24:03,069 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000005
2016-02-16 13:24:03,112 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000005
2016-02-16 13:24:03,135 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 117.2 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:06,143 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 118.7 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:09,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 14725 for container-id container_1455610525380_0004_01_000001: 118.7 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used
2016-02-16 13:24:09,273 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_1455610525380_0004_01_000001 succeeded 
2016-02-16 13:24:09,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
2016-02-16 13:24:09,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1455610525380_0004_01_000001
2016-02-16 13:24:09,290 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004/container_1455610525380_0004_01_000001
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserOPERATION=Container Finished - SucceededTARGET=ContainerImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000001
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1455610525380_0004_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1455610525380_0004_01_000001 from application application_1455610525380_0004
2016-02-16 13:24:09,291 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1455610525380_0004
2016-02-16 13:24:10,083 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1455610525380_0004_000001 (auth:SIMPLE)
2016-02-16 13:24:10,086 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1455610525380_0004_01_000001
2016-02-16 13:24:10,086 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hduserIP=172.31.41.153OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1455610525380_0004CONTAINERID=container_1455610525380_0004_01_000001
2016-02-16 13:24:10,087 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1455610525380_0004_01_000001, container_1455610525380_0004_01_000005]
2016-02-16 13:24:10,087 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1455610525380_0004
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1455610525380_0004 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1455610525380_0004, with delay of 10800 seconds
2016-02-16 13:24:10,088 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1455610525380_0004
2016-02-16 13:24:12,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455610525380_0004_01_000001
2016-02-16 13:45:26,936 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /usr/local/hadoop/logs/userlogs/application_1455610525380_0001
2016-02-16 13:50:46,142 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /usr/local/hadoop/logs/userlogs/application_1455610525380_0002
2016-02-16 13:51:55,525 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /usr/local/hadoop/logs/userlogs/application_1455610525380_0003
2016-02-16 15:36:21,382 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM
2016-02-16 15:36:21,402 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8042
2016-02-16 15:36:21,403 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Applications still running : [application_1455610525380_0004]
2016-02-16 15:36:21,403 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Waiting for Applications to be Finished
2016-02-16 15:36:25,404 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Done waiting for Applications to be Finished. Still alive: [application_1455610525380_0004]
2016-02-16 15:36:25,404 INFO org.apache.hadoop.ipc.Server: Stopping server on 57583
2016-02-16 15:36:25,406 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2016-02-16 15:36:25,407 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 57583
Highlighted

Re: wordcount not completed with Yarn exception ...

Mentor

@luc tiber can you run service check on YARN, make sure the nodemanagers work correctly. Please go to the failed job logs and drill down to the individual attempt logs that failed. I cannot see real issue here.

Highlighted

Re: wordcount not completed with Yarn exception ...

Explorer

@Artem Ervits

I tried to run service check as follow and get an unexpected message (see below); Am I missing something

thanks!

yarn rmadmin -checkHealth 6597

Cannot run -checkHealth when ResourceManager HA is not enabled

Highlighted

Re: wordcount not completed with Yarn exception ...

Mentor

@luc tiber did you enable HA for resource manager? If so, you probably misconfigured it. Doublecheck your steps with our documentation for setting up resource manager high availability. As soon as you configure it, run service checks again.

Highlighted

Re: wordcount not completed with Yarn exception ...

Explorer

No I did not. it is a single node testing env... so HA...

yet I noticed the following warnings in yarn node manager log file (might be related)

1- org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: NodeManager configured with 8 G physical memory allocated to containers, which is more than 80% of the total physical memory available (992.5 M). Thrashing might happen.

This on the resource manager log

2- org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue, it is likely set too low. skipping enforcement to allow at least one application to start

org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent is insufficient to start a single application in queue for user, it is likely set too low. skipping enforcement to allow at least one application to start

So I had a look into my capacity-scheduler.xml file but could not really figure out if it´s related or not

Thank!

LT

Highlighted

Re: wordcount not completed with Yarn exception ...

Mentor

@luc tiber did you assign queues and configure allocation? You might have assigned less resources than necessary to a queue. Take a look at this http://hortonworks.com/hadoop-tutorial/configuring-yarn-capacity-scheduler-ambari/

Don't have an account?
Coming from Hortonworks? Activate your account here