Problem in YARN (distcp) - sometime gives error message "Operation not permitted"

sphinxid — Fri, 16 Sep 2022 09:16:06 GMT

Hello,

I'm not sure why this error message sometime appeared while doing some map/reduce job including distcp.

I would really appreciate some help to pointing out this problem.

# below is the error msg.

14/12/19 07:07:24 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs://##########/data/events/video/20141212], targetPath=hdfs://kishimoto/data/events/video, targetPathExists=true, preserveRawXattrs=false}
14/12/19 07:07:24 INFO client.RMProxy: Connecting to ResourceManager at XXXXXXXXXX/10.66.27.131:8032
14/12/19 07:07:25 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
14/12/19 07:07:25 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
14/12/19 07:07:25 INFO client.RMProxy: Connecting to ResourceManager at XXXXXXXXXX/10.66.27.131:8032
14/12/19 07:07:25 INFO mapreduce.JobSubmitter: number of splits:6
14/12/19 07:07:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1418971058569_0022
14/12/19 07:07:26 INFO impl.YarnClientImpl: Submitted application application_1418971058569_0022
14/12/19 07:07:26 INFO mapreduce.Job: The url to track the job: http://XXXXXXXXXX:8088/proxy/application_1418971058569_0022/
14/12/19 07:07:26 INFO tools.DistCp: DistCp job-id: job_1418971058569_0022
14/12/19 07:07:26 INFO mapreduce.Job: Running job: job_1418971058569_0022
14/12/19 07:07:41 INFO mapreduce.Job: Job job_1418971058569_0022 running in uber mode : false
14/12/19 07:07:41 INFO mapreduce.Job: map 0% reduce 0%
14/12/19 07:07:48 INFO mapreduce.Job: Task Id : attempt_1418971058569_0022_m_000002_0, Status : FAILED
Application application_1418971058569_0022 initialization failed (exitCode=1) with output: main : command provided 0
main : user is yarn
main : requested yarn user is centos
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:228)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:642)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:434)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1063)
at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:162)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:197)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:721)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:717)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:717)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createDir(ContainerLocalizer.java:383)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:369)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:129)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:347)

14/12/19 07:07:52 INFO mapreduce.Job: map 33% reduce 0%
14/12/19 07:07:53 INFO mapreduce.Job: map 83% reduce 0%
14/12/19 07:07:56 INFO mapreduce.Job: Task Id : attempt_1418971058569_0022_m_000002_1, Status : FAILED
Application application_1418971058569_0022 initialization failed (exitCode=1) with output: main : command provided 0
main : user is yarn
main : requested yarn user is centos
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:228)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:642)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:434)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1063)
at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:162)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:197)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:721)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:717)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:717)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createDir(ContainerLocalizer.java:383)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:369)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:129)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:347)

14/12/19 07:08:02 INFO mapreduce.Job: map 100% reduce 0%
14/12/19 07:08:03 INFO mapreduce.Job: Job job_1418971058569_0022 completed successfully
14/12/19 07:08:03 INFO mapreduce.Job: Counters: 34
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=665190
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=62556154
HDFS: Number of bytes written=62552365
HDFS: Number of read operations=105
HDFS: Number of large read operations=0
HDFS: Number of write operations=23
Job Counters
Failed map tasks=2
Launched map tasks=8
Other local map tasks=8
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=43559
Total vcore-seconds taken by all map tasks=87118
Total megabyte-seconds taken by all map tasks=44604416
Map-Reduce Framework
Map input records=6
Map output records=0
Input split bytes=708
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=100
CPU time spent (ms)=6890
Physical memory (bytes) snapshot=1396334592
Virtual memory (bytes) snapshot=10175578112
Total committed heap usage (bytes)=3029336064
File Input Format Counters
Bytes Read=3081
File Output Format Counters
Bytes Written=0
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESCOPIED=62552365
BYTESEXPECTED=62552365
COPY=6

Re: Problem in YARN (distcp) - sometime gives error message "Operation not permitted"

sphinxid — Sat, 20 Dec 2014 06:51:18 GMT

ok i found out the problem. I use CDH 5.2.1.

In my case, it happened when I use YARN to manage Impala resources. (default config)

Then those error msg appeared.

question Problem in YARN (distcp) - sometime gives error message "Operation not permitted" in Support Questions

Problem in YARN (distcp) - sometime gives error message "Operation not permitted"

Re: Problem in YARN (distcp) - sometime gives error message "Operation not permitted"