Error: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for user/myname/.cm/distcp-staging/2025-09-21-05-14-47-c4f9/intermediate.1 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:447) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:152) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:133) at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:3566) at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:3360) at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:3336) at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2899) at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:2938) at com.cloudera.enterprise.distcp.util.DistCpUtils.sortListing(DistCpUtils.java:427) at com.cloudera.enterprise.distcp.mapred.StatusReducer.lambda$deleteMissing$1(StatusReducer.java:152) at com.cloudera.enterprise.distcp.mapred.StatusReducerProgress.track(StatusReducerProgress.java:211) at com.cloudera.enterprise.distcp.mapred.StatusReducerProgress.trackSortSourceListing(StatusReducerProgress.java:223) at com.cloudera.enterprise.distcp.mapred.StatusReducer.deleteMissing(StatusReducer.java:151) at com.cloudera.enterprise.distcp.mapred.StatusReducer.cleanup(StatusReducer.java:89) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
This is the error I am getting at first I suspected if i don't have enough space in my local dir but all the dir have enough space, should not be permission issue as it only fail sometimes most of the time replication is successful but this issue persist atleast once a week. Can someone help where should i look next.
