After upgrading from 5.2.1 to 5.3.1 we are seeing EXT4 errors on our HDFS/Yarm Manager nodes:
DatanodeRegistration(10.15.129.105, datanodeUuid=d2f178f8-1399-4330-92b9-c627b0a04804, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=cluster15;nsid=601399481;c=0):Got exception while serving BP-447169353-10.15.129.109-1415094554153:blk_1110216188_36477305 to /10.15.129.103:56258
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-447169353-10.15.129.109-1415094554153:blk_1110216188_36477305
The upgrade did not throw any exceptions and "hdfs fsck /" is clean. How can I flush these out of the system? Maybe these are files that did not have time to replicate properly before the upgrade?