Here's the error from the BDR job Running in the DR system. Below that I've run an fsck on the file on the source system to show that it does exist on the source and has the same block number as listed in the error. I've removed ip addresses and I've removed the actual file name and replaced with "filename" ERROR /path/filename org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1508298398-ipaddress-1406065203774:blk_2079737512_1100628731148 file=filename at org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1040) at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1023) at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1002) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:895) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:954) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.cloudera.enterprise.distcp.util.ThrottledInputStream.read(ThrottledInputStream.java:77) at com.cloudera.enterprise.distcp.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:371) at com.cloudera.enterprise.distcp.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:345) at com.cloudera.enterprise.distcp.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:161) at com.cloudera.enterprise.distcp.util.RetriableCommand.execute(RetriableCommand.java:87) at com.cloudera.enterprise.distcp.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:617) at com.cloudera.enterprise.distcp.mapred.CopyMapper.map(CopyMapper.java:454) at com.cloudera.enterprise.distcp.mapred.CopyMapper.map(CopyMapper.java:69) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop Here's the file on the source system - hdfs fsck filename -files -blocks -locations Connecting to namenode via http://prodservername:50070 FSCK started by hdfs (auth:KERBEROS_SSL) from /serveripaddress for path filename at Mon Aug 27 19:39:53 EDT 2018 filename 995352 bytes, 1 block(s): OK 0. BP-1508298398-ipaddress-1406065203774:blk_2079737512_1100628731148 len=995352 Live_repl=3 [DatanodeInfoWithStorage[ipaddress:1004,DS-00246250-eef8-4c03-8ef7-c898594f960b,DISK], DatanodeInfoWithStorage[ipaddress:1004,DS-297b0420-a2a1-4418-8691-3ef9a374cc51,DISK], DatanodeInfoWithStorage[ipaddress:1004,DS-0ae9f985-a12a-4871-991b-d2e8017c4c4b,DISK]]
... View more
