2017-10-06 10:28:09,371 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(810)) - Start request for container_e40_1507297721771_0001_01_000008 by user winnie 2017-10-06 10:28:09,375 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(850)) - Creating a new application reference for app application_1507297721771_0001 2017-10-06 10:28:09,384 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1507297721771_0001 transitioned from NEW to INITING 2017-10-06 10:28:09,401 INFO application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_e40_1507297721771_0001_01_000008 to application application_1507297721771_0001 2017-10-06 10:28:09,405 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application application_1507297721771_0001 transitioned from INITING to RUNNING 2017-10-06 10:28:09,406 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000008 transitioned from NEW to LOCALIZING 2017-10-06 10:28:09,406 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event CONTAINER_INIT for appId application_1507297721771_0001 2017-10-06 10:28:09,406 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(184)) - Initializing container container_e40_1507297721771_0001_01_000008 2017-10-06 10:28:09,406 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(287)) - Initializing container container_e40_1507297721771_0001_01_000008 2017-10-06 10:28:09,406 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event APPLICATION_INIT for appId application_1507297721771_0001 2017-10-06 10:28:09,406 INFO containermanager.AuxServices (AuxServices.java:handle(219)) - Got APPLICATION_INIT for service mapreduce_shuffle 2017-10-06 10:28:09,407 INFO mapred.ShuffleHandler (ShuffleHandler.java:addJobToken(681)) - Added token for job_1507297721771_0001 2017-10-06 10:28:09,407 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://vlmazgrpmaster.fisdev.local:8020/tmp/hive/winnie/_tez_session_dir/3c76b620-643d-423e-b5f3-2b713c2f0cb7/hive-hcatalog-core.jar transitioned from INIT to DOWNLOADING 2017-10-06 10:28:09,407 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://vlmazgrpmaster.fisdev.local:8020/tmp/hive/winnie/_tez_session_dir/3c76b620-643d-423e-b5f3-2b713c2f0cb7/.tez/application_1507297721771_0001/tez-conf.pb transitioned from INIT to DOWNLOADING 2017-10-06 10:28:09,407 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(712)) - Created localizer for container_e40_1507297721771_0001_01_000008 2017-10-06 10:28:09,421 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1194)) - Writing credentials to the nmPrivate file /appl/hadoop/yarn/local/nmPrivate/container_e40_1507297721771_0001_01_000008.tokens. Credentials list: 2017-10-06 10:28:09,438 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /appl/hadoop/yarn/local/usercache, should be rwxr-xr-x, actual value = rwxrwxrwx 2017-10-06 10:28:09,443 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/db/hadoop/yarn/local/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x 2017-10-06 10:28:09,447 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/local/opt/hadoop/yarn/local/nmPrivate, should be rwx------, actual value = rwx-w---- 2017-10-06 10:28:09,447 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:getInitializedLocalDirs(1410)) - Attempting to initialize /appl/hadoop/yarn/local 2017-10-06 10:28:09,452 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:initializeLocalDir(1277)) - Could not set permissions for local dir /appl/hadoop/yarn/local/usercache EPERM: Operation not permitted at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:999) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1273) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:141) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1111) 2017-10-06 10:28:09,452 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(1134)) - Localizer failed org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not set permissions for local dir /appl/hadoop/yarn/local/usercache at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1278) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:141) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1111) Caused by: EPERM: Operation not permitted at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:999) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1273) ... 3 more 2017-10-06 10:28:09,452 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000008 transitioned from LOCALIZING to LOCALIZATION_FAILED 2017-10-06 10:28:09,454 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000008 transitioned from LOCALIZATION_FAILED to DONE 2017-10-06 10:28:09,454 INFO application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_e40_1507297721771_0001_01_000008 from application application_1507297721771_0001 2017-10-06 10:28:09,454 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(512)) - Considering container container_e40_1507297721771_0001_01_000008 for log-aggregation 2017-10-06 10:28:09,455 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event CONTAINER_STOP for appId application_1507297721771_0001 2017-10-06 10:28:09,455 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(190)) - Stopping container container_e40_1507297721771_0001_01_000008 2017-10-06 10:28:09,455 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(293)) - Stopping container container_e40_1507297721771_0001_01_000008 2017-10-06 10:28:09,739 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(810)) - Start request for container_e40_1507297721771_0001_01_000012 by user winnie 2017-10-06 10:28:09,748 INFO application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_e40_1507297721771_0001_01_000012 to application application_1507297721771_0001 2017-10-06 10:28:09,748 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000012 transitioned from NEW to LOCALIZING 2017-10-06 10:28:09,748 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event CONTAINER_INIT for appId application_1507297721771_0001 2017-10-06 10:28:09,748 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(184)) - Initializing container container_e40_1507297721771_0001_01_000012 2017-10-06 10:28:09,749 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(287)) - Initializing container container_e40_1507297721771_0001_01_000012 2017-10-06 10:28:09,749 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event APPLICATION_INIT for appId application_1507297721771_0001 2017-10-06 10:28:09,749 INFO containermanager.AuxServices (AuxServices.java:handle(219)) - Got APPLICATION_INIT for service mapreduce_shuffle 2017-10-06 10:28:09,749 INFO mapred.ShuffleHandler (ShuffleHandler.java:addJobToken(681)) - Added token for job_1507297721771_0001 2017-10-06 10:28:09,750 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://vlmazgrpmaster.fisdev.local:8020/tmp/hive/winnie/_tez_session_dir/3c76b620-643d-423e-b5f3-2b713c2f0cb7/hive-hcatalog-core.jar transitioned from INIT to DOWNLOADING 2017-10-06 10:28:09,750 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://vlmazgrpmaster.fisdev.local:8020/tmp/hive/winnie/_tez_session_dir/3c76b620-643d-423e-b5f3-2b713c2f0cb7/.tez/application_1507297721771_0001/tez-conf.pb transitioned from INIT to DOWNLOADING 2017-10-06 10:28:09,754 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(712)) - Created localizer for container_e40_1507297721771_0001_01_000012 2017-10-06 10:28:09,755 WARN fs.LocalDirAllocator$AllocatorPerContext (LocalDirAllocator.java:createPath(353)) - Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /usr/local/opt/hadoop/yarn/local/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:81) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:485) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1101) 2017-10-06 10:28:09,755 WARN fs.LocalDirAllocator$AllocatorPerContext (LocalDirAllocator.java:createPath(353)) - Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /usr/db/hadoop/yarn/local/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:81) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:485) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1101) 2017-10-06 10:28:09,765 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1194)) - Writing credentials to the nmPrivate file /appl/hadoop/yarn/local/nmPrivate/container_e40_1507297721771_0001_01_000012.tokens. Credentials list: 2017-10-06 10:28:09,782 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /appl/hadoop/yarn/local/usercache, should be rwxr-xr-x, actual value = rwxrwxrwx 2017-10-06 10:28:09,786 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/db/hadoop/yarn/local/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x 2017-10-06 10:28:09,790 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/local/opt/hadoop/yarn/local/nmPrivate, should be rwx------, actual value = rwx-w---- 2017-10-06 10:28:09,790 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:getInitializedLocalDirs(1410)) - Attempting to initialize /appl/hadoop/yarn/local 2017-10-06 10:28:09,795 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:initializeLocalDir(1277)) - Could not set permissions for local dir /appl/hadoop/yarn/local/usercache EPERM: Operation not permitted at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:999) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1273) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:141) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1111) 2017-10-06 10:28:09,795 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(1134)) - Localizer failed org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not set permissions for local dir /appl/hadoop/yarn/local/usercache at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1278) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:141) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1111) Caused by: EPERM: Operation not permitted at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:999) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1273) ... 3 more 2017-10-06 10:28:09,796 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000012 transitioned from LOCALIZING to LOCALIZATION_FAILED 2017-10-06 10:28:09,798 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000012 transitioned from LOCALIZATION_FAILED to DONE 2017-10-06 10:28:09,799 INFO application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_e40_1507297721771_0001_01_000012 from application application_1507297721771_0001 2017-10-06 10:28:09,799 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(512)) - Considering container container_e40_1507297721771_0001_01_000012 for log-aggregation 2017-10-06 10:28:09,799 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event CONTAINER_STOP for appId application_1507297721771_0001 2017-10-06 10:28:09,799 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(190)) - Stopping container container_e40_1507297721771_0001_01_000012 2017-10-06 10:28:09,800 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(293)) - Stopping container container_e40_1507297721771_0001_01_000012 2017-10-06 10:28:10,065 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInternal(810)) - Start request for container_e40_1507297721771_0001_01_000013 by user winnie 2017-10-06 10:28:10,072 INFO application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding container_e40_1507297721771_0001_01_000013 to application application_1507297721771_0001 2017-10-06 10:28:10,073 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000013 transitioned from NEW to LOCALIZING 2017-10-06 10:28:10,073 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event CONTAINER_INIT for appId application_1507297721771_0001 2017-10-06 10:28:10,073 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(184)) - Initializing container container_e40_1507297721771_0001_01_000013 2017-10-06 10:28:10,074 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(287)) - Initializing container container_e40_1507297721771_0001_01_000013 2017-10-06 10:28:10,074 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event APPLICATION_INIT for appId application_1507297721771_0001 2017-10-06 10:28:10,074 INFO containermanager.AuxServices (AuxServices.java:handle(219)) - Got APPLICATION_INIT for service mapreduce_shuffle 2017-10-06 10:28:10,074 INFO mapred.ShuffleHandler (ShuffleHandler.java:addJobToken(681)) - Added token for job_1507297721771_0001 2017-10-06 10:28:10,075 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://vlmazgrpmaster.fisdev.local:8020/tmp/hive/winnie/_tez_session_dir/3c76b620-643d-423e-b5f3-2b713c2f0cb7/hive-hcatalog-core.jar transitioned from INIT to DOWNLOADING 2017-10-06 10:28:10,075 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://vlmazgrpmaster.fisdev.local:8020/tmp/hive/winnie/_tez_session_dir/3c76b620-643d-423e-b5f3-2b713c2f0cb7/.tez/application_1507297721771_0001/tez-conf.pb transitioned from INIT to DOWNLOADING 2017-10-06 10:28:10,075 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(712)) - Created localizer for container_e40_1507297721771_0001_01_000013 2017-10-06 10:28:10,076 WARN fs.LocalDirAllocator$AllocatorPerContext (LocalDirAllocator.java:createPath(353)) - Disk Error Exception: org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /usr/db/hadoop/yarn/local/nmPrivate at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:81) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:350) at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:412) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:116) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:485) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1101) 2017-10-06 10:28:10,086 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(1194)) - Writing credentials to the nmPrivate file /appl/hadoop/yarn/local/nmPrivate/container_e40_1507297721771_0001_01_000013.tokens. Credentials list: 2017-10-06 10:28:10,108 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /appl/hadoop/yarn/local/usercache, should be rwxr-xr-x, actual value = rwxrwxrwx 2017-10-06 10:28:10,114 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/db/hadoop/yarn/local/usercache, should be rwxr-xr-x, actual value = rwxrwxr-x 2017-10-06 10:28:10,119 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/local/opt/hadoop/yarn/local/nmPrivate, should be rwx------, actual value = rwx-w---- 2017-10-06 10:28:10,119 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:getInitializedLocalDirs(1410)) - Attempting to initialize /appl/hadoop/yarn/local 2017-10-06 10:28:10,124 WARN localizer.ResourceLocalizationService (ResourceLocalizationService.java:initializeLocalDir(1277)) - Could not set permissions for local dir /appl/hadoop/yarn/local/usercache EPERM: Operation not permitted at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:999) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1273) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:141) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1111) 2017-10-06 10:28:10,124 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:run(1134)) - Localizer failed org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not set permissions for local dir /appl/hadoop/yarn/local/usercache at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1278) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.getInitializedLocalDirs(ResourceLocalizationService.java:1411) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.access$800(ResourceLocalizationService.java:141) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1111) Caused by: EPERM: Operation not permitted at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:756) at org.apache.hadoop.fs.DelegateToFileSystem.setPermission(DelegateToFileSystem.java:211) at org.apache.hadoop.fs.FilterFs.setPermission(FilterFs.java:252) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:1003) at org.apache.hadoop.fs.FileContext$11.next(FileContext.java:999) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.setPermission(FileContext.java:999) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.initializeLocalDir(ResourceLocalizationService.java:1273) ... 3 more 2017-10-06 10:28:10,125 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000013 transitioned from LOCALIZING to LOCALIZATION_FAILED 2017-10-06 10:28:10,128 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_e40_1507297721771_0001_01_000013 transitioned from LOCALIZATION_FAILED to DONE 2017-10-06 10:28:10,128 INFO application.ApplicationImpl (ApplicationImpl.java:transition(347)) - Removing container_e40_1507297721771_0001_01_000013 from application application_1507297721771_0001 2017-10-06 10:28:10,128 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:startContainerLogAggregation(512)) - Considering container container_e40_1507297721771_0001_01_000013 for log-aggregation 2017-10-06 10:28:10,128 INFO containermanager.AuxServices (AuxServices.java:handle(215)) - Got event CONTAINER_STOP for appId application_1507297721771_0001 2017-10-06 10:28:10,128 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(190)) - Stopping container container_e40_1507297721771_0001_01_000013 2017-10-06 10:28:10,128 INFO yarn.YarnShuffleService (YarnShuffleService.java:stopContainer(293)) - Stopping container container_e40_1507297721771_0001_01_000013 2017-10-06 10:28:10,131 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(553)) - Removed completed containers from NM context: [container_e40_1507297721771_0001_01_000008] 2017-10-06 10:28:10,134 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:stopContainerInternal(960)) - Stopping container with container Id: container_e40_1507297721771_0001_01_000013 2017-10-06 10:28:11,132 INFO nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:removeOrTrackCompletedContainersFromContext(553)) - Removed completed containers from NM context: [container_e40_1507297721771_0001_01_000012, container_e40_1507297721771_0001_01_000013] 2017-10-06 10:28:11,134 WARN containermanager.ContainerManagerImpl (ContainerManagerImpl.java:handle(1070)) - Event EventType: KILL_CONTAINER sent to absent container container_e40_1507297721771_0001_01_000014 2017-10-06 10:28:11,837 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(425)) - Stopping resource-monitoring for container_e40_1507297721771_0001_01_000008 2017-10-06 10:28:11,837 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(425)) - Stopping resource-monitoring for container_e40_1507297721771_0001_01_000012 2017-10-06 10:28:11,837 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(425)) - Stopping resource-monitoring for container_e40_1507297721771_0001_01_000013