Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Archives of Support Questions (Read Only)
This is an archived board for historical reference. Information and links may no longer be available or relevant
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.
- Cloudera Community
- Board Archive
- Archives of Support Questions (Read Only)
- Re: Map Reduce Application is not getting submitte...
Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
Map Reduce Application is not getting submitted to YARN.
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
-
Apache YARN
Expert Contributor
Created 05-26-2016 09:30 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have 5 node cluster. 2 master and 3 datanodes. all machine has 16GB memory. also there are almost 6GB free memory available on each machine. I have a requirement to export a hbase table to hdfs. and hence i am running the following command.
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot CUTOFF1-SNAPSHOT -copy-to hdfs://prodcluster/hbase-export
After issuing the above command i am getting the following error.
[main] impl.YarnClientImpl: Application submission is not finished, submitted application application_1459775087681_0596 is still in NEW_SAVING
and the above line is getting printed in console at every seconds.
Please help!!
1 ACCEPTED SOLUTION
Super Guru
Created 05-26-2016 11:08 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Raja Ray
Can you check application logs for any error ? If possible can you get screenshot of Yarn webUI attached here.
Also from below link i see restarting yarn helps -
5 REPLIES 5
Guru
Created 05-26-2016 09:43 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Raja Ray,
Do you have logs from YARN node manager?
Super Guru
Created 05-26-2016 11:08 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Raja Ray
Can you check application logs for any error ? If possible can you get screenshot of Yarn webUI attached here.
Also from below link i see restarting yarn helps -
Expert Contributor
Created 05-26-2016 12:53 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Pierre Villard ,
I restarted App TimeLine Server and ResourceManager and that solved the problem. however, here is log of yarn node manager.
2016-05-26 05:47:55,939 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initializing container container_1464255636652_0012_01_000001 2016-05-26 05:47:55,943 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,943 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-server-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,943 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/netty-all-4.0.23.Final.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/protobuf-java-2.5.0.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/htrace-core-3.1.0-incubating.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-client-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/metrics-core-2.2.0.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/zookeeper-3.4.6.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-common-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/guava-12.0.1.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.split transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.splitmetainfo transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.xml transitioned from INIT to DOWNLOADING 2016-05-26 05:47:55,944 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_1464255636652_0012_01_000001 2016-05-26 05:47:55,945 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1191)) - Writing credentials to the nmPrivate file /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000001.tokens. Credentials list: 2016-05-26 05:47:55,967 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(610)) - Initializing user hdfs 2016-05-26 05:47:55,968 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(117)) - Copying from /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000001.tokens to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001.tokens 2016-05-26 05:47:55,968 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(124)) - Localizer CWD set to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 = file:/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 2016-05-26 05:47:56,008 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4052/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,033 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-server-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4053/hbase-server-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,052 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/netty-all-4.0.23.Final.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4054/netty-all-4.0.23.Final.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,068 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/protobuf-java-2.5.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4055/protobuf-java-2.5.0.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,093 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4056/hadoop-common-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,111 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/htrace-core-3.1.0-incubating.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4057/htrace-core-3.1.0-incubating.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,135 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-client-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4058/hbase-client-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,150 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/metrics-core-2.2.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4059/metrics-core-2.2.0.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,167 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/zookeeper-3.4.6.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4060/zookeeper-3.4.6.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,192 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4061/hbase-protocol-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,208 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-common-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4062/hbase-common-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,223 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4063/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,242 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/libjars/guava-12.0.1.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4064/guava-12.0.1.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,260 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.split(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/10/job.split) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,276 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.splitmetainfo(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/11/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,302 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/12/job.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,318 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0012/job.xml(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:47:56,318 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-05-26 05:47:56,336 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from LOCALIZED to RUNNING 2016-05-26 05:47:56,338 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(268)) - launchContainer: [bash, /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001/default_container_executor.sh] 2016-05-26 05:47:57,647 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(375)) - Starting resource-monitoring for container_1464255636652_0012_01_000001 2016-05-26 05:47:57,657 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 142.0 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used 2016-05-26 05:48:00,686 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 341.4 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used 2016-05-26 05:48:03,324 INFO ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0012_000001 (auth:SIMPLE) 2016-05-26 05:48:03,330 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(816)) - Start request for container_1464255636652_0012_01_000002 by user hdfs 2016-05-26 05:48:03,331 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs IP=10.1.10.204 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1464255636652_0012 CONTAINERID=container_1464255636652_0012_01_000002 2016-05-26 05:48:03,331 INFO application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_1464255636652_0012_01_000002 to application application_1464255636652_0012 2016-05-26 05:48:03,331 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from NEW to LOCALIZING 2016-05-26 05:48:03,332 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1464255636652_0012 2016-05-26 05:48:03,332 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initializing container container_1464255636652_0012_01_000002 2016-05-26 05:48:03,332 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event APPLICATION_INIT for appId application_1464255636652_0012 2016-05-26 05:48:03,332 INFO containermanager.AuxServices (AuxServices.java:handle(200)) - Got APPLICATION_INIT for service mapreduce_shuffle 2016-05-26 05:48:03,332 INFO mapred.ShuffleHandler (ShuffleHandler.java:addJobToken(671)) - Added token for job_1464255636652_0012 2016-05-26 05:48:03,332 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:03,332 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.xml transitioned from INIT to DOWNLOADING 2016-05-26 05:48:03,333 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_1464255636652_0012_01_000002 2016-05-26 05:48:03,334 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1191)) - Writing credentials to the nmPrivate file /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000002.tokens. Credentials list: 2016-05-26 05:48:03,380 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(610)) - Initializing user hdfs 2016-05-26 05:48:03,381 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(117)) - Copying from /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0012_01_000002.tokens to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002.tokens 2016-05-26 05:48:03,381 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(124)) - Localizer CWD set to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 = file:/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 2016-05-26 05:48:03,435 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/14/job.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:03,456 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0012/job.xml(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/filecache/15/job.xml) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:03,456 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from LOCALIZING to LOCALIZED 2016-05-26 05:48:03,478 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from LOCALIZED to RUNNING 2016-05-26 05:48:03,481 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(268)) - launchContainer: [bash, /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002/default_container_executor.sh] 2016-05-26 05:48:03,686 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(375)) - Starting resource-monitoring for container_1464255636652_0012_01_000002 2016-05-26 05:48:03,701 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 395.1 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:03,723 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26129 for container-id container_1464255636652_0012_01_000002: 43.3 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used 2016-05-26 05:48:06,474 INFO ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0012_000001 (auth:SIMPLE) 2016-05-26 05:48:06,478 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_1464255636652_0012_01_000002 2016-05-26 05:48:06,478 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs IP=10.1.10.204 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1464255636652_0012 CONTAINERID=container_1464255636652_0012_01_000002 2016-05-26 05:48:06,478 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from RUNNING to KILLING 2016-05-26 05:48:06,478 INFO launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(371)) - Cleaning up container container_1464255636652_0012_01_000002 2016-05-26 05:48:06,502 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(224)) - Exit code from container container_1464255636652_0012_01_000002 is : 143 2016-05-26 05:48:06,509 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL 2016-05-26 05:48:06,509 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1464255636652_0012 CONTAINERID=container_1464255636652_0012_01_000002 2016-05-26 05:48:06,510 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002 2016-05-26 05:48:06,510 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000002 2016-05-26 05:48:06,511 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE 2016-05-26 05:48:06,511 INFO application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_1464255636652_0012_01_000002 from application application_1464255636652_0012 2016-05-26 05:48:06,511 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(547)) - Considering container container_1464255636652_0012_01_000002 for log-aggregation 2016-05-26 05:48:06,511 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1464255636652_0012 2016-05-26 05:48:06,511 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_1464255636652_0012_01_000002 2016-05-26 05:48:06,724 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(390)) - Stopping resource-monitoring for container_1464255636652_0012_01_000002 2016-05-26 05:48:06,733 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 411.9 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:09,746 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 412.8 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:12,755 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 25938 for container-id container_1464255636652_0012_01_000001: 412.8 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:12,912 INFO launcher.ContainerLaunch (ContainerLaunch.java:call(347)) - Container container_1464255636652_0012_01_000001 succeeded 2016-05-26 05:48:12,913 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS 2016-05-26 05:48:12,913 INFO launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(371)) - Cleaning up container container_1464255636652_0012_01_000001 2016-05-26 05:48:12,935 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001 2016-05-26 05:48:12,935 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012/container_1464255636652_0012_01_000001 2016-05-26 05:48:12,935 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs OPERATION=Container Finished - SucceededTARGET=ContainerImpl RESULT=SUCCESS APPID=application_1464255636652_0012 CONTAINERID=container_1464255636652_0012_01_000001 2016-05-26 05:48:12,936 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0012_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE 2016-05-26 05:48:12,936 INFO application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_1464255636652_0012_01_000001 from application application_1464255636652_0012 2016-05-26 05:48:12,936 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(547)) - Considering container container_1464255636652_0012_01_000001 for log-aggregation 2016-05-26 05:48:12,936 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1464255636652_0012 2016-05-26 05:48:12,936 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_1464255636652_0012_01_000001 2016-05-26 05:48:13,495 INFO ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0012_000001 (auth:SIMPLE) 2016-05-26 05:48:13,499 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_1464255636652_0012_01_000001 2016-05-26 05:48:13,500 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs IP=10.1.10.20 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1464255636652_0012 CONTAINERID=container_1464255636652_0012_01_000001 2016-05-26 05:48:13,500 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(529)) - Removed completed containers from NM context: [container_1464255636652_0012_01_000001] 2016-05-26 05:48:13,501 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0012 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-05-26 05:48:13,501 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event APPLICATION_STOP for appId application_1464255636652_0012 2016-05-26 05:48:13,501 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopApplication(170)) - Stopping application application_1464255636652_0012 2016-05-26 05:48:13,501 INFO shuffle.ExternalShuffleBlockResolver (ExternalShuffleBlockResolver.java:applicationRemoved(206)) - Application application_1464255636652_0012 removed, cleanupLocalDirs = false 2016-05-26 05:48:13,501 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0012 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-05-26 05:48:13,501 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just finished : application_1464255636652_0012 2016-05-26 05:48:13,502 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 2016-05-26 05:48:13,503 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0012 2016-05-26 05:48:13,513 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggregation(602)) - Uploading logs for container container_1464255636652_0012_01_000001. Current good log dirs are /var/log/hadoop/yarn/log,/hadoop/hdfs1/hadoop/yarn/log 2016-05-26 05:48:13,514 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggregation(602)) - Uploading logs for container container_1464255636652_0012_01_000002. Current good log dirs are /var/log/hadoop/yarn/log,/hadoop/hdfs1/hadoop/yarn/log 2016-05-26 05:48:13,514 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/directory.info 2016-05-26 05:48:13,514 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/launch_container.sh 2016-05-26 05:48:13,514 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/stderr 2016-05-26 05:48:13,515 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/stdout 2016-05-26 05:48:13,515 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000001/syslog 2016-05-26 05:48:13,516 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/stderr 2016-05-26 05:48:13,516 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/directory.info 2016-05-26 05:48:13,516 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/syslog 2016-05-26 05:48:13,516 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/launch_container.sh 2016-05-26 05:48:13,516 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012/container_1464255636652_0012_01_000002/stdout 2016-05-26 05:48:13,550 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0012 2016-05-26 05:48:13,551 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0012 2016-05-26 05:48:14,520 INFO ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0013_000001 (auth:SIMPLE) 2016-05-26 05:48:14,522 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(816)) - Start request for container_1464255636652_0013_01_000001 by user hdfs 2016-05-26 05:48:14,523 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(856)) - Creating a new application reference for app application_1464255636652_0013 2016-05-26 05:48:14,523 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from NEW to INITING 2016-05-26 05:48:14,523 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs IP=10.1.10.20 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1464255636652_0013 CONTAINERID=container_1464255636652_0013_01_000001 2016-05-26 05:48:14,524 WARN logaggregation.LogAggregationService (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users. 2016-05-26 05:48:14,525 WARN logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:<init>(190)) - rollingMonitorInterval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this application is finished. 2016-05-26 05:48:14,543 INFO application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_1464255636652_0013_01_000001 to application application_1464255636652_0013 2016-05-26 05:48:14,543 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from INITING to RUNNING 2016-05-26 05:48:14,543 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from NEW to LOCALIZING 2016-05-26 05:48:14,544 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_INIT for appId application_1464255636652_0013 2016-05-26 05:48:14,544 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initializing container container_1464255636652_0013_01_000001 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-server-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/netty-all-4.0.23.Final.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/protobuf-java-2.5.0.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/htrace-core-3.1.0-incubating.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-client-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/metrics-core-2.2.0.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/zookeeper-3.4.6.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-common-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/guava-12.0.1.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.split transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.splitmetainfo transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.jar transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.xml transitioned from INIT to DOWNLOADING 2016-05-26 05:48:14,544 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_1464255636652_0013_01_000001 2016-05-26 05:48:14,545 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1191)) - Writing credentials to the nmPrivate file /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0013_01_000001.tokens. Credentials list: 2016-05-26 05:48:14,568 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(610)) - Initializing user hdfs 2016-05-26 05:48:14,569 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(117)) - Copying from /hadoop/hdfs1/hadoop/yarn/local/nmPrivate/container_1464255636652_0013_01_000001.tokens to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001.tokens 2016-05-26 05:48:14,569 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(124)) - Localizer CWD set to /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013 = file:/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013 2016-05-26 05:48:14,613 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4065/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,638 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-server-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4066/hbase-server-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,657 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/netty-all-4.0.23.Final.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4067/netty-all-4.0.23.Final.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,675 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/protobuf-java-2.5.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4068/protobuf-java-2.5.0.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,698 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hadoop-common-2.7.1.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4069/hadoop-common-2.7.1.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,716 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/htrace-core-3.1.0-incubating.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4070/htrace-core-3.1.0-incubating.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,734 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-client-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4071/hbase-client-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,748 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/metrics-core-2.2.0.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4072/metrics-core-2.2.0.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,765 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/zookeeper-3.4.6.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4073/zookeeper-3.4.6.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,790 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-protocol-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4074/hbase-protocol-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,810 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-common-1.1.2.2.4.0.0-169.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4075/hbase-common-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,828 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/filecache/4076/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,847 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster/user/hdfs/.staging/job_1464255636652_0013/libjars/guava-12.0.1.jar(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/filecache/4077/guava-12.0.1.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,864 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.split(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/10/job.split) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,880 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.splitmetainfo(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/11/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,912 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.jar(->/var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/12/job.jar) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,930 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://aimprodcluster:8020/user/hdfs/.staging/job_1464255636652_0013/job.xml(->/hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED 2016-05-26 05:48:14,930 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from LOCALIZING to LOCALIZED 2016-05-26 05:48:14,955 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from LOCALIZED to RUNNING 2016-05-26 05:48:14,959 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExecutor(268)) - launchContainer: [bash, /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001/default_container_executor.sh] 2016-05-26 05:48:15,757 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(375)) - Starting resource-monitoring for container_1464255636652_0013_01_000001 2016-05-26 05:48:15,757 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(390)) - Stopping resource-monitoring for container_1464255636652_0012_01_000001 2016-05-26 05:48:15,787 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 108.4 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used 2016-05-26 05:48:18,801 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 316.7 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used 2016-05-26 05:48:21,809 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 349.4 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:24,823 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 384.5 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:27,832 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 410.3 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:30,841 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(464)) - Memory usage of ProcessTree 26444 for container-id container_1464255636652_0013_01_000001: 410.3 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used 2016-05-26 05:48:32,057 INFO launcher.ContainerLaunch (ContainerLaunch.java:call(347)) - Container container_1464255636652_0013_01_000001 succeeded 2016-05-26 05:48:32,057 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS 2016-05-26 05:48:32,057 INFO launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(371)) - Cleaning up container container_1464255636652_0013_01_000001 2016-05-26 05:48:32,079 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001 2016-05-26 05:48:32,079 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013/container_1464255636652_0013_01_000001 2016-05-26 05:48:32,079 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs OPERATION=Container Finished - SucceededTARGET=ContainerImpl RESULT=SUCCESS APPID=application_1464255636652_0013 CONTAINERID=container_1464255636652_0013_01_000001 2016-05-26 05:48:32,080 INFO container.ContainerImpl (ContainerImpl.java:handle(1131)) - Container container_1464255636652_0013_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE 2016-05-26 05:48:32,080 INFO application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_1464255636652_0013_01_000001 from application application_1464255636652_0013 2016-05-26 05:48:32,080 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(547)) - Considering container container_1464255636652_0013_01_000001 for log-aggregation 2016-05-26 05:48:32,080 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId application_1464255636652_0013 2016-05-26 05:48:32,080 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(189)) - Stopping container container_1464255636652_0013_01_000001 2016-05-26 05:48:32,550 INFO ipc.Server (Server.java:saslProcess(1386)) - Auth successful for appattempt_1464255636652_0013_000001 (auth:SIMPLE) 2016-05-26 05:48:32,553 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(966)) - Stopping container with container Id: container_1464255636652_0013_01_000001 2016-05-26 05:48:32,553 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=hdfs IP=10.1.10.20 OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1464255636652_0013 CONTAINERID=container_1464255636652_0013_01_000001 2016-05-26 05:48:32,556 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(529)) - Removed completed containers from NM context: [container_1464255636652_0013_01_000001] 2016-05-26 05:48:32,556 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2016-05-26 05:48:32,557 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /var/log/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013 2016-05-26 05:48:32,557 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(458)) - Deleting absolute path : /hadoop/hdfs1/hadoop/yarn/local/usercache/hdfs/appcache/application_1464255636652_0013 2016-05-26 05:48:32,557 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event APPLICATION_STOP for appId application_1464255636652_0013 2016-05-26 05:48:32,557 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopApplication(170)) - Stopping application application_1464255636652_0013 2016-05-26 05:48:32,557 INFO shuffle.ExternalShuffleBlockResolver (ExternalShuffleBlockResolver.java:applicationRemoved(206)) - Application application_1464255636652_0013 removed, cleanupLocalDirs = false 2016-05-26 05:48:32,557 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1464255636652_0013 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2016-05-26 05:48:32,557 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregation(555)) - Application just finished : application_1464255636652_0013 2016-05-26 05:48:32,569 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggregation(602)) - Uploading logs for container container_1464255636652_0013_01_000001. Current good log dirs are /var/log/hadoop/yarn/log,/hadoop/hdfs1/hadoop/yarn/log 2016-05-26 05:48:32,569 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/stdout 2016-05-26 05:48:32,570 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/launch_container.sh 2016-05-26 05:48:32,570 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/directory.info 2016-05-26 05:48:32,570 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/syslog 2016-05-26 05:48:32,570 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013/container_1464255636652_0013_01_000001/stderr 2016-05-26 05:48:32,606 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /var/log/hadoop/yarn/log/application_1464255636652_0013 2016-05-26 05:48:32,607 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(467)) - Deleting path : /hadoop/hdfs1/hadoop/yarn/log/application_1464255636652_0013 2016-05-26 05:48:33,841 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(390)) - Stopping resource-monitoring for container_1464255636652_0013_01_000001
Thanks
Raja ray
Expert Contributor
Created 05-26-2016 04:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks. Restarting App timeline server worked for me.
Guru
Created 03-24-2017 02:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Does "hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot" use ATS?
Maybe restarting ResrouceManager fixed the issue?
Announcements
Community Announcements
What's New @ Cloudera
What's New @ Cloudera
Community Announcements
Community Announcements
