Hi,
someone meet the error before. when submit job to yarn.
hdfs://nameservice1/tmp/wc.py changed on src filesystem (expected 1512455039858, was 1512455040707)
I dig into the source-code, found that, Yarn try to download the resource. and it will check the lastmodificationtime with the timestamp.
org.apache.hadoop.yarn.util.FSDownload
private Path copy( Path sCopy , Path dstdir ) throws IOException {
FileSystem sourceFs = sCopy .getFileSystem( conf );
Path dCopy = new Path ( dstdir , "tmp_" + sCopy .getName());
FileStatus sStat = sourceFs .getFileStatus( sCopy );
if ( sStat .getModificationTime() != resource .getTimestamp()) {
throw new IOException ( "Resource " + sCopy +
" changed on src filesystem (expected " + resource .getTimestamp() +
", was " + sStat .getModificationTime());
}
if ( resource .getVisibility() == LocalResourceVisibility. PUBLIC ) {
if (!isPublic( sourceFs , sCopy , sStat , statCache )) {
throw new IOException ( "Resource " + sCopy +
" is not publicly accessable and as such cannot be part of the" +
" public cache." );
}
}
FileUtil .copy( sourceFs , sStat , FileSystem .getLocal( conf ), dCopy , false ,
true , conf );
return dCopy ;
}
... View more
