> Do you get a full strack trace in the Namenode log at the time of the error in the datanode? No. I showed the entire log msg. that was visible. I checked NN stdout, and did not see a stacktrace. > Have all these files with missing blocks got replication factor of 1 or have they a replication factor 3? repl factor is 3, but the problem files did not reach that level due to competing acivity. The DN in question had to be bounced due to a failed disk (it has 36 8TB disks), apparently at an inopportune moment. I think the excessive RBW files (up to 1 year old) are not the cause since I moved most of them away, restarted DN, and there was no change. Current problem summary: finalized blocks/replicas in a DN are reported as missing, and "RemoteException in offerService" WARN appears in DN log. More info: I found that of the 4400 block/replicas missing cluster-wide, there are 3500 spread across 36 disks on one DN under /finalized/, unevenly: count, disk 2 09 2 11 2 21 3 31 4 24 4 34 5 20 5 32 6 01 7 13 7 19 7 33 10 28 15 05 139 23 154 04 159 27 160 25 171 26 280 17 hence, I am unable to place blame on one disk. This DN has 14M blk files, so only a tiny percentage are affected. It sure seems like an uncaught exception caused the block report from this DN to be incomplete.
... View more
