Map Reduce Application is not getting submitted to YARN.

[#1]

I have 5 node cluster. 2 master and 3 datanodes. all machine has 16GB memory. also there are almost 6GB free memory available on each machine. I have a requirement to export a hbase table to hdfs. and hence i am running the following command.

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot CUTOFF1-SNAPSHOT -copy-to hdfs://prodcluster/hbase-export

After issuing the above command i am getting the following error.

[main] impl.YarnClientImpl: Application submission is not finished, submitted application application_1459775087681_0596 is still in NEW_SAVING

and the above line is getting printed in console at every seconds.

Please help!!

[#2]

Hi @Raja Ray,

Do you have logs from YARN node manager?

[#3 · Accepted solution]

@Raja Ray

Can you check application logs for any error ? If possible can you get screenshot of Yarn webUI attached here.

Also from below link i see restarting yarn helps -

http://stackoverflow.com/questions/33496491/high-availability-jobs-not-getting-submitted-immediately...

[#4]

Hi @Pierre Villard ,

I restarted App TimeLine Server and ResourceManager and that solved the problem. however, here is log of yarn node manager.

2016-05-26 05:47:55,939 INFO  yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initializing container container_1464255636652_0012_01_000001
2016-05-26 05:47:55,943 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,943 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-server-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,943 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/netty-all-4.0.23.Final.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/protobuf-java-2.5.0.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/htrace-core-3.1.0-incubating.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-client-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/metrics-core-2.2.0.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/zookeeper-3.4.6.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-common-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/guava-12.0.1.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.split transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.splitmetainfo transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.xml transitioned from INIT to DOWNLOADING
2016-05-26 05:47:55,944 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_1464255636652_0012_01_000001
2016-05-26 05:47:55,945 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1191)) - Writing credentials to the nmPrivate file /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000001.tokens. Credentials list: 
2016-05-26 05:47:55,967 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(610)) - Initializing user hdfs
2016-05-26 05:47:55,968 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(117)) - Copying from /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000001.tokens to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001.tokens
2016-05-26 05:47:55,968 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(124)) - Localizer CWD set to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 = file:/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012
2016-05-26 05:47:56,008 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4052/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,033 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-server-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4053/hbase-server-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,052 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/netty-all-4.0.23.Final.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4054/netty-all-4.0.23.Final.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,068 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/protobuf-java-2.5.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4055/protobuf-java-2.5.0.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,093 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4056/hadoop-common-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,111 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/htrace-core-3.1.0-incubating.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4057/htrace-core-3.1.0-incubating.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,135 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-client-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4058/hbase-client-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,150 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/metrics-core-2.2.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4059/metrics-core-2.2.0.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,167 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/zookeeper-3.4.6.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4060/zookeeper-3.4.6.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,192 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4061/hbase-protocol-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,208 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-common-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4062/hbase-common-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,223 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4063/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,242 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/guava-12.0.1.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4064/guava-12.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,260 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.split(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/10/job.split) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,276 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.splitmetainfo(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/11/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,302 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/12/job.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,318 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.xml(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:47:56,318 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from LOCALIZING to LOCALIZED
2016-05-26 05:47:56,336 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from LOCALIZED to RUNNING
2016-05-26 05:47:56,338 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(268)) - launchContainer: [bash, /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001/default_container_executor.sh]
2016-05-26 05:47:57,647 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(375)) - Starting resource-monitoring for container_1464255636652_0012_01_000001
2016-05-26 05:47:57,657 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 142.0 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used
2016-05-26 05:48:00,686 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 341.4 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used
2016-05-26 05:48:03,324 INFO  ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0012_000001 (auth:SIMPLE)
2016-05-26 05:48:03,330 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(816)) - Start request for container_1464255636652_0012_01_000002 by user hdfs
2016-05-26 05:48:03,331 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	IP=10.1.10.204	OPERATION=Start Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1464255636652_0012	CONTAINERID=container_1464255636652_0012_01_000002
2016-05-26 05:48:03,331 INFO  application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_1464255636652_0012_01_000002 to application application_1464255636652_0012
2016-05-26 05:48:03,331 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from NEW to LOCALIZING
2016-05-26 05:48:03,332 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1464255636652_0012
2016-05-26 05:48:03,332 INFO  yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initializing container container_1464255636652_0012_01_000002
2016-05-26 05:48:03,332 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event APPLICATION_INIT for appId application_1464255636652_0012
2016-05-26 05:48:03,332 INFO  containermanager.AuxServices (AuxServices.java:handle(200)) - Got APPLICATION_INIT for service mapreduce_shuffle
2016-05-26 05:48:03,332 INFO  mapred.ShuffleHandler (ShuffleHandler.java:addJobToken(671)) - Added token for job_1464255636652_0012
2016-05-26 05:48:03,332 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:03,332 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.xml transitioned from INIT to DOWNLOADING
2016-05-26 05:48:03,333 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_1464255636652_0012_01_000002
2016-05-26 05:48:03,334 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1191)) - Writing credentials to the nmPrivate file /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000002.tokens. Credentials list: 
2016-05-26 05:48:03,380 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(610)) - Initializing user hdfs
2016-05-26 05:48:03,381 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(117)) - Copying from /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000002.tokens to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002.tokens
2016-05-26 05:48:03,381 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(124)) - Localizer CWD set to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 = file:/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012
2016-05-26 05:48:03,435 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/14/job.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:03,456 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.xml(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/15/job.xml) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:03,456 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from LOCALIZING to LOCALIZED
2016-05-26 05:48:03,478 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from LOCALIZED to RUNNING
2016-05-26 05:48:03,481 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(268)) - launchContainer: [bash, /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002/default_container_executor.sh]
2016-05-26 05:48:03,686 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(375)) - Starting resource-monitoring for container_1464255636652_0012_01_000002
2016-05-26 05:48:03,701 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 395.1 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:03,723 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26129 for container-id container_1464255636652_0012_01_000002: 43.3 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used
2016-05-26 05:48:06,474 INFO  ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0012_000001 (auth:SIMPLE)
2016-05-26 05:48:06,478 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_1464255636652_0012_01_000002
2016-05-26 05:48:06,478 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	IP=10.1.10.204	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1464255636652_0012	CONTAINERID=container_1464255636652_0012_01_000002
2016-05-26 05:48:06,478 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from RUNNING to KILLING
2016-05-26 05:48:06,478 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(371)) - Cleaning up container container_1464255636652_0012_01_000002
2016-05-26 05:48:06,502 WARN  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(224)) - Exit code from container container_1464255636652_0012_01_000002 is : 143
2016-05-26 05:48:06,509 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2016-05-26 05:48:06,509 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	OPERATION=Container Finished - Killed	TARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1464255636652_0012	CONTAINERID=container_1464255636652_0012_01_000002
2016-05-26 05:48:06,510 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002
2016-05-26 05:48:06,510 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002
2016-05-26 05:48:06,511 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2016-05-26 05:48:06,511 INFO  application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_1464255636652_0012_01_000002 from application application_1464255636652_0012
2016-05-26 05:48:06,511 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(547)) - Considering container container_1464255636652_0012_01_000002 for log-aggregation
2016-05-26 05:48:06,511 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1464255636652_0012
2016-05-26 05:48:06,511 INFO  yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_1464255636652_0012_01_000002
2016-05-26 05:48:06,724 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(390)) - Stopping resource-monitoring for container_1464255636652_0012_01_000002
2016-05-26 05:48:06,733 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 411.9 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:09,746 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 412.8 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:12,755 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 412.8 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:12,912 INFO  launcher.ContainerLaunch (ContainerLaunch.java:call(347)) - Container container_1464255636652_0012_01_000001 succeeded 
2016-05-26 05:48:12,913 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
2016-05-26 05:48:12,913 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(371)) - Cleaning up container container_1464255636652_0012_01_000001
2016-05-26 05:48:12,935 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001
2016-05-26 05:48:12,935 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001
2016-05-26 05:48:12,935 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	OPERATION=Container Finished - SucceededTARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1464255636652_0012	CONTAINERID=container_1464255636652_0012_01_000001
2016-05-26 05:48:12,936 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2016-05-26 05:48:12,936 INFO  application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_1464255636652_0012_01_000001 from application application_1464255636652_0012
2016-05-26 05:48:12,936 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(547)) - Considering container container_1464255636652_0012_01_000001 for log-aggregation
2016-05-26 05:48:12,936 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1464255636652_0012
2016-05-26 05:48:12,936 INFO  yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_1464255636652_0012_01_000001
2016-05-26 05:48:13,495 INFO  ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0012_000001 (auth:SIMPLE)
2016-05-26 05:48:13,499 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_1464255636652_0012_01_000001
2016-05-26 05:48:13,500 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	IP=10.1.10.20	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1464255636652_0012	CONTAINERID=container_1464255636652_0012_01_000001
2016-05-26 05:48:13,500 INFO  nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(529)) - Removed completed containers from NM context: [container_1464255636652_0012_01_000001]
2016-05-26 05:48:13,501 INFO  application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0012 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2016-05-26 05:48:13,501 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event APPLICATION_STOP for appId application_1464255636652_0012
2016-05-26 05:48:13,501 INFO  yarn.YarnShuffleService (YarnShuffleService.java:stopApplication(170)) - Stopping application application_1464255636652_0012
2016-05-26 05:48:13,501 INFO  shuffle.ExternalShuffleBlockResolver (ExternalShuffleBlockResolver.java:applicationRemoved(206)) - Application application_1464255636652_0012 removed, cleanupLocalDirs = false
2016-05-26 05:48:13,501 INFO  application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0012 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2016-05-26 05:48:13,501 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just finished : application_1464255636652_0012
2016-05-26 05:48:13,502 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012
2016-05-26 05:48:13,503 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012
2016-05-26 05:48:13,513 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggregation(602)) - Uploading logs for container container_1464255636652_0012_01_000001. Current good log dirs are /var/log/hadoop/yarn/log,/hadoop/hdfs1/hadoop/yarn/log
2016-05-26 05:48:13,514 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggregation(602)) - Uploading logs for container container_1464255636652_0012_01_000002. Current good log dirs are /var/log/hadoop/yarn/log,/hadoop/hdfs1/hadoop/yarn/log
2016-05-26 05:48:13,514 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/directory.info
2016-05-26 05:48:13,514 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/launch_container.sh
2016-05-26 05:48:13,514 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/stderr
2016-05-26 05:48:13,515 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/stdout
2016-05-26 05:48:13,515 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/syslog
2016-05-26 05:48:13,516 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/stderr
2016-05-26 05:48:13,516 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/directory.info
2016-05-26 05:48:13,516 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/syslog
2016-05-26 05:48:13,516 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/launch_container.sh
2016-05-26 05:48:13,516 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/stdout
2016-05-26 05:48:13,550 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012
2016-05-26 05:48:13,551 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012
2016-05-26 05:48:14,520 INFO  ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0013_000001 (auth:SIMPLE)
2016-05-26 05:48:14,522 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(816)) - Start request for container_1464255636652_0013_01_000001 by user hdfs
2016-05-26 05:48:14,523 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(856)) - Creating a new application reference for app application_1464255636652_0013
2016-05-26 05:48:14,523 INFO  application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from NEW to INITING
2016-05-26 05:48:14,523 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	IP=10.1.10.20	OPERATION=Start Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1464255636652_0013	CONTAINERID=container_1464255636652_0013_01_000001
2016-05-26 05:48:14,524 WARN  logaggregation.LogAggregationService (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users.
2016-05-26 05:48:14,525 WARN  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:<init>(190)) - rollingMonitorInterval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this application is finished.
2016-05-26 05:48:14,543 INFO  application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_1464255636652_0013_01_000001 to application application_1464255636652_0013
2016-05-26 05:48:14,543 INFO  application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from INITING to RUNNING
2016-05-26 05:48:14,543 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from NEW to LOCALIZING
2016-05-26 05:48:14,544 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1464255636652_0013
2016-05-26 05:48:14,544 INFO  yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initializing container container_1464255636652_0013_01_000001
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-server-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/netty-all-4.0.23.Final.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/protobuf-java-2.5.0.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/htrace-core-3.1.0-incubating.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-client-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/metrics-core-2.2.0.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/zookeeper-3.4.6.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-common-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/guava-12.0.1.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.split transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.splitmetainfo transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.jar transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.xml transitioned from INIT to DOWNLOADING
2016-05-26 05:48:14,544 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_1464255636652_0013_01_000001
2016-05-26 05:48:14,545 INFO  localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1191)) - Writing credentials to the nmPrivate file /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0013_01_000001.tokens. Credentials list: 
2016-05-26 05:48:14,568 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(610)) - Initializing user hdfs
2016-05-26 05:48:14,569 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(117)) - Copying from /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0013_01_000001.tokens to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001.tokens
2016-05-26 05:48:14,569 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(124)) - Localizer CWD set to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013 = file:/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013
2016-05-26 05:48:14,613 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4065/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,638 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-server-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4066/hbase-server-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,657 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/netty-all-4.0.23.Final.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4067/netty-all-4.0.23.Final.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,675 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/protobuf-java-2.5.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4068/protobuf-java-2.5.0.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,698 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4069/hadoop-common-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,716 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/htrace-core-3.1.0-incubating.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4070/htrace-core-3.1.0-incubating.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,734 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-client-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4071/hbase-client-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,748 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/metrics-core-2.2.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4072/metrics-core-2.2.0.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,765 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/zookeeper-3.4.6.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4073/zookeeper-3.4.6.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,790 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4074/hbase-protocol-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,810 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-common-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4075/hbase-common-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,828 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4076/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,847 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/guava-12.0.1.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4077/guava-12.0.1.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,864 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.split(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/10/job.split) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,880 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.splitmetainfo(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/11/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,912 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/12/job.jar) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,930 INFO  localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.xml(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED
2016-05-26 05:48:14,930 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from LOCALIZING to LOCALIZED
2016-05-26 05:48:14,955 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from LOCALIZED to RUNNING
2016-05-26 05:48:14,959 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(268)) - launchContainer: [bash, /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001/default_container_executor.sh]
2016-05-26 05:48:15,757 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(375)) - Starting resource-monitoring for container_1464255636652_0013_01_000001
2016-05-26 05:48:15,757 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(390)) - Stopping resource-monitoring for container_1464255636652_0012_01_000001
2016-05-26 05:48:15,787 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 108.4 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used
2016-05-26 05:48:18,801 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 316.7 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used
2016-05-26 05:48:21,809 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 349.4 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:24,823 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 384.5 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:27,832 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 410.3 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:30,841 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 410.3 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used
2016-05-26 05:48:32,057 INFO  launcher.ContainerLaunch (ContainerLaunch.java:call(347)) - Container container_1464255636652_0013_01_000001 succeeded 
2016-05-26 05:48:32,057 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
2016-05-26 05:48:32,057 INFO  launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(371)) - Cleaning up container container_1464255636652_0013_01_000001
2016-05-26 05:48:32,079 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001
2016-05-26 05:48:32,079 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001
2016-05-26 05:48:32,079 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	OPERATION=Container Finished - SucceededTARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_1464255636652_0013	CONTAINERID=container_1464255636652_0013_01_000001
2016-05-26 05:48:32,080 INFO  container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2016-05-26 05:48:32,080 INFO  application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_1464255636652_0013_01_000001 from application application_1464255636652_0013
2016-05-26 05:48:32,080 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(547)) - Considering container container_1464255636652_0013_01_000001 for log-aggregation
2016-05-26 05:48:32,080 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1464255636652_0013
2016-05-26 05:48:32,080 INFO  yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_1464255636652_0013_01_000001
2016-05-26 05:48:32,550 INFO  ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0013_000001 (auth:SIMPLE)
2016-05-26 05:48:32,553 INFO  containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_1464255636652_0013_01_000001
2016-05-26 05:48:32,553 INFO  nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs	IP=10.1.10.20	OPERATION=Stop Container Request	TARGET=ContainerManageImpl	RESULT=SUCCESS	APPID=application_1464255636652_0013	CONTAINERID=container_1464255636652_0013_01_000001
2016-05-26 05:48:32,556 INFO  nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(529)) - Removed completed containers from NM context: [container_1464255636652_0013_01_000001]
2016-05-26 05:48:32,556 INFO  application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2016-05-26 05:48:32,557 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013
2016-05-26 05:48:32,557 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013
2016-05-26 05:48:32,557 INFO  containermanager.AuxServices (AuxServices.java:handle(196)) - Got event APPLICATION_STOP for appId application_1464255636652_0013
2016-05-26 05:48:32,557 INFO  yarn.YarnShuffleService (YarnShuffleService.java:stopApplication(170)) - Stopping application application_1464255636652_0013
2016-05-26 05:48:32,557 INFO  shuffle.ExternalShuffleBlockResolver (ExternalShuffleBlockResolver.java:applicationRemoved(206)) - Application application_1464255636652_0013 removed, cleanupLocalDirs = false
2016-05-26 05:48:32,557 INFO  application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2016-05-26 05:48:32,557 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just finished : application_1464255636652_0013
2016-05-26 05:48:32,569 INFO  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggregation(602)) - Uploading logs for container container_1464255636652_0013_01_000001. Current good log dirs are /var/log/hadoop/yarn/log,/hadoop/hdfs1/hadoop/yarn/log
2016-05-26 05:48:32,569 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/stdout
2016-05-26 05:48:32,570 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/launch_container.sh
2016-05-26 05:48:32,570 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/directory.info
2016-05-26 05:48:32,570 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/syslog
2016-05-26 05:48:32,570 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/stderr
2016-05-26 05:48:32,606 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0013
2016-05-26 05:48:32,607 INFO  nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013
2016-05-26 05:48:33,841 INFO  monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(390)) - Stopping resource-monitoring for container_1464255636652_0013_01_000001

Thanks

Raja ray

[#5]

Thanks. Restarting App timeline server worked for me.

[#6]

Does "hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot" use ATS?

Maybe restarting ResrouceManager fixed the issue?