https://community.cloudera.com/t5/Archives-of-Support-Questions/I-would-like-to-know-if-there-are-any-data-duplication-i-e/m-p/105066#M33431 <P><A rel="user" href="https://community.cloudera.com/users/9842/mpandit.html" nodeid="9842">@milind pandit</A></P><P>There is no direct utility to find this. </P><P>The files with different names but same content will have have same checksum. Using checksum option of hdfs , we can verify the same.</P><P>For example:</P><PRE># hdfs dfs -ls /tmp/tst Found 6 items -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/okay -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/pass -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/pass3 -rw-r--r-- 3 hdfs hdfs 1064 2016-06-29 21:46 /tmp/tst/pre -rw-r--r-- 3 hdfs hdfs 1064 2016-06-29 21:46 /tmp/tst/pro -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/word</PRE> <PRE># hdfs dfs -checksum /tmp/tst/okay /tmp/tst/okay MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 # hdfs dfs -checksum /tmp/tst/pass /tmp/tst/pass MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 # hdfs dfs -checksum /tmp/tst/pre /tmp/tst/pre MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d # hdfs dfs -checksum /tmp/tst/pro /tmp/tst/pro MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d </PRE><P>From the above, the files "/tmp/tst/okay" and "/tmp/tst/pass" are holding same content, but the filenames are different. You can see from above that both files have same checksum. Similarly for "/tmp/tst/pro" and "/tmp/tst/pre".</P><P>To check the checksum of files on a folder ( in this case "/tmp/tst" ) , following can be done:</P><PRE># hdfs dfs -checksum /tmp/tst/* /tmp/tst/okay MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pass MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pass3 MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pre MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d /tmp/tst/pro MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d /tmp/tst/word MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7</PRE><P>Also, you can use "hdfs find" to make a large search:</P><PRE># hdfs dfs -checksum `hdfs dfs -find /tmp -print` </PRE><P>The above command will list checksum of all the files. </P><P>You can also run with "sort and uniq " as :</P><PRE>hdfs dfs -checksum `hdfs dfs -find /tmp -print` | sort | uniq -c | awk '{print $2,$4}'</PRE> Thu, 30 Jun 2016 11:54:48 GMT PARTOMIA 2016-06-30T11:54:48Z https://community.cloudera.com/t5/Archives-of-Support-Questions/I-would-like-to-know-if-there-are-any-data-duplication-i-e/m-p/105065#M33430 Thu, 30 Jun 2016 04:16:16 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/I-would-like-to-know-if-there-are-any-data-duplication-i-e/m-p/105065#M33430 mpandit 2016-06-30T04:16:16Z https://community.cloudera.com/t5/Archives-of-Support-Questions/I-would-like-to-know-if-there-are-any-data-duplication-i-e/m-p/105066#M33431 <P><A rel="user" href="https://community.cloudera.com/users/9842/mpandit.html" nodeid="9842">@milind pandit</A></P><P>There is no direct utility to find this. </P><P>The files with different names but same content will have have same checksum. Using checksum option of hdfs , we can verify the same.</P><P>For example:</P><PRE># hdfs dfs -ls /tmp/tst Found 6 items -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/okay -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/pass -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/pass3 -rw-r--r-- 3 hdfs hdfs 1064 2016-06-29 21:46 /tmp/tst/pre -rw-r--r-- 3 hdfs hdfs 1064 2016-06-29 21:46 /tmp/tst/pro -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/word</PRE> <PRE># hdfs dfs -checksum /tmp/tst/okay /tmp/tst/okay MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 # hdfs dfs -checksum /tmp/tst/pass /tmp/tst/pass MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 # hdfs dfs -checksum /tmp/tst/pre /tmp/tst/pre MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d # hdfs dfs -checksum /tmp/tst/pro /tmp/tst/pro MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d </PRE><P>From the above, the files "/tmp/tst/okay" and "/tmp/tst/pass" are holding same content, but the filenames are different. You can see from above that both files have same checksum. Similarly for "/tmp/tst/pro" and "/tmp/tst/pre".</P><P>To check the checksum of files on a folder ( in this case "/tmp/tst" ) , following can be done:</P><PRE># hdfs dfs -checksum /tmp/tst/* /tmp/tst/okay MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pass MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pass3 MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pre MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d /tmp/tst/pro MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d /tmp/tst/word MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7</PRE><P>Also, you can use "hdfs find" to make a large search:</P><PRE># hdfs dfs -checksum `hdfs dfs -find /tmp -print` </PRE><P>The above command will list checksum of all the files. </P><P>You can also run with "sort and uniq " as :</P><PRE>hdfs dfs -checksum `hdfs dfs -find /tmp -print` | sort | uniq -c | awk '{print $2,$4}'</PRE> Thu, 30 Jun 2016 11:54:48 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/I-would-like-to-know-if-there-are-any-data-duplication-i-e/m-p/105066#M33431 PARTOMIA 2016-06-30T11:54:48Z