08-09-2018 07:16 AM
As part of our cloudera BDR backup & restore validation,we use the below commad to verify the back up and restored files are same.
hdfs dfs -count /data
before start the replication schedule . my /data directory in source cluster contains 6982 directories and 10,887 files. Please see the result of the hdfs count command
[user@example ~]$ hdfs dfs -count /data
6982 10,887 11897305288 /data
[user@example~]$ hdfs dfs -ls -R /data | wc -l
we had run replication(via distcp command line)maually, due to some space crunch on the remote server the distcp job was failed. then we run below command to check the hdfs count
[user@example tmp]$ hdfs dfs -count /data
6982 21756 11940958360 /data
[user@example tmp]$ hdfs dfs -ls -R /data | wc -l
There was a devation in the file count before the operation,almost the file count increased double. However
ls -R result giving the actual count (6982 +10,887).
Ideally the output of hdfs dfs -count command should returns with 10,887 files and 6982 directories.
What could be the reason for this inconsistent result? We did restart the cluster suspecting some chache but despite that the counts mentioned above was consitent.
Thanks in advance,
08-11-2018 04:00 AM
08-23-2018 04:47 AM
Hi, I think it is related to the snapshots or hidden directories. Maybe the distcp is preparing a snapshot, and as it failed, it left these temporary objects in HDFS.