Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎05-08-2018

hadoop -count returning wrong result

Hi All,

 

As part of our cloudera BDR backup & restore validation,we use the below commad to  verify the back up and restored files are same.

 

hdfs dfs -count /data

 

before start the replication schedule . my /data directory in source cluster contains 6982 directories and 10,887 files. Please see the result of the hdfs count command 

[user@example ~]$ hdfs dfs -count /data
6982 10,887 11897305288 /data

             &

[user@example~]$ hdfs dfs -ls -R /data | wc -l
17869

 

 we had run replication(via distcp command line)maually, due to some space crunch on the remote server the distcp job was failed. then we run below command to check the hdfs count 

 

[user@example tmp]$ hdfs dfs -count /data
6982 21756 11940958360 /data

 

[user@example tmp]$ hdfs dfs -ls -R /data | wc -l
17869

 

There was a devation in the file count before the operation,almost the file count increased double. However

ls -R result giving the actual count (6982 +10,887).

 

Ideally the output of hdfs dfs -count command should returns with 10,887 files and 6982 directories.

 

What could be the reason for this inconsistent result? We did restart the cluster suspecting some chache but despite that the counts mentioned above was consitent.

 

Thanks in advance,

Kathik

Champion
Posts: 739
Registered: ‎05-16-2016

Re: hadoop -count returning wrong result

If you have enterprise version you could download the disk usage report and verify it and see which folder has the most of the files .

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_dg_disk_usage_reports.html
Announcements