Reply
Explorer
Posts: 14
Registered: ‎06-06-2017

hdfs://nameservice1/tmp/wc.py changed on src filesystem (expected 1512455039858, was 1512455040707)

[ Edited ]

Hi,

 

someone meet the error before. when submit job to yarn. 

 

hdfs://nameservice1/tmp/wc.py changed on src filesystem (expected 1512455039858, was 1512455040707)

 

I dig into the source-code, found that, Yarn try to download the resource. and it will check the lastmodificationtime with the timestamp.

 

org.apache.hadoop.yarn.util.FSDownload

 

private Path copy(Path sCopy, Path dstdir) throws IOException {

    FileSystem sourceFs = sCopy.getFileSystem(conf);

    Path dCopy = new Path(dstdir, "tmp_"+sCopy.getName());

    FileStatus sStat = sourceFs.getFileStatus(sCopy);

    if (sStat.getModificationTime() != resource.getTimestamp()) {

      throw new IOException("Resource " + sCopy +

          " changed on src filesystem (expected " + resource.getTimestamp() +

          ", was " + sStat.getModificationTime());

    }

    if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {

      if (!isPublic(sourceFs, sCopy, sStat, statCache)) {

        throw new IOException("Resource " + sCopy +

            " is not publicly accessable and as such cannot be part of the" +

            " public cache.");

      }

    }

 

    FileUtil.copy(sourceFs, sStat, FileSystem.getLocal(conf), dCopy, false,

        true, conf);

    return dCopy;

  }

Announcements