Support Questions

Find answers, ask questions, and share your expertise

RedHat and HDFS report different values for "du"

avatar
Explorer

I copied a large folder structure from HDFS to RedHat using copyToLocal.  While it looked successful, i want to validate it was copied correctly by checking the size of the data in HDFS and in RedHat.  I'm using "du" but my numbers are still off. 

 

I run the following on RH: "du -s -b <PATH>"

I run the following on HDFS: "hadoop fs -du -s <PATH>"

 

I noticed that RH reports 101 bytes for empty folders while HDFS (CDH5.5.2) reports 0 bytes for empty folders.  So my question is, how to I validate the entire directory of data was fully transferred?

 

Thanks,

1 ACCEPTED SOLUTION

avatar
Explorer
As an alternative solution, I am comparing the two file structures for the same number of folders and files.

View solution in original post

1 REPLY 1

avatar
Explorer
As an alternative solution, I am comparing the two file structures for the same number of folders and files.