Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

hdfs://nameservice1/tmp/wc.py changed on src filesystem (expected 1512455039858, was 1512455040707)

hdfs://nameservice1/tmp/wc.py changed on src filesystem (expected 1512455039858, was 1512455040707)

Explorer

Hi,

 

someone meet the error before. when submit job to yarn. 

 

hdfs://nameservice1/tmp/wc.py changed on src filesystem (expected 1512455039858, was 1512455040707)

 

I dig into the source-code, found that, Yarn try to download the resource. and it will check the lastmodificationtime with the timestamp.

 

org.apache.hadoop.yarn.util.FSDownload

 

private Path copy(Path sCopy, Path dstdir) throws IOException {

    FileSystem sourceFs = sCopy.getFileSystem(conf);

    Path dCopy = new Path(dstdir, "tmp_"+sCopy.getName());

    FileStatus sStat = sourceFs.getFileStatus(sCopy);

    if (sStat.getModificationTime() != resource.getTimestamp()) {

      throw new IOException("Resource " + sCopy +

          " changed on src filesystem (expected " + resource.getTimestamp() +

          ", was " + sStat.getModificationTime());

    }

    if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {

      if (!isPublic(sourceFs, sCopy, sStat, statCache)) {

        throw new IOException("Resource " + sCopy +

            " is not publicly accessable and as such cannot be part of the" +

            " public cache.");

      }

    }

 

    FileUtil.copy(sourceFs, sStat, FileSystem.getLocal(conf), dCopy, false,

        true, conf);

    return dCopy;

  }