I got it resolved, the issue was related to permissions on the bucket by the certian user. ... View more

@milind pandit This is exactly the kind of thing that Tags are for. When an entity, for example a hive table, is tagged, it will show up when you view all entities associated with that tag. With the new Atlas/Ranger integration, you can create security policies that only apply to hive tables or even hive table columns that have been tagged with that tag. This allows you control access to the hive table or hive table columns just by adding or removing the tag or adding/removing use groups to whom the tag based policy applies. For example, using Atlas and Ranger, you can easily keep track of Data Sets classified as PII and control and audit access to these data sets. ... View more

@milind pandit There is no direct utility to find this. The files with different names but same content will have have same checksum. Using checksum option of hdfs , we can verify the same. For example: # hdfs dfs -ls /tmp/tst Found 6 items -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/okay -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/pass -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/pass3 -rw-r--r-- 3 hdfs hdfs 1064 2016-06-29 21:46 /tmp/tst/pre -rw-r--r-- 3 hdfs hdfs 1064 2016-06-29 21:46 /tmp/tst/pro -rw-r--r-- 3 hdfs hdfs 2044 2016-06-29 21:46 /tmp/tst/word # hdfs dfs -checksum /tmp/tst/okay /tmp/tst/okay MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 # hdfs dfs -checksum /tmp/tst/pass /tmp/tst/pass MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 # hdfs dfs -checksum /tmp/tst/pre /tmp/tst/pre MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d # hdfs dfs -checksum /tmp/tst/pro /tmp/tst/pro MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d From the above, the files "/tmp/tst/okay" and "/tmp/tst/pass" are holding same content, but the filenames are different. You can see from above that both files have same checksum. Similarly for "/tmp/tst/pro" and "/tmp/tst/pre". To check the checksum of files on a folder ( in this case "/tmp/tst" ) , following can be done: # hdfs dfs -checksum /tmp/tst/* /tmp/tst/okay MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pass MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pass3 MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 /tmp/tst/pre MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d /tmp/tst/pro MD5-of-0MD5-of-512CRC32C 000002000000000000000000690e462cbf52c9c399fb7c0bcacef01d /tmp/tst/word MD5-of-0MD5-of-512CRC32C 000002000000000000000000b1be3e03929521974dc321f9e7f27cc7 Also, you can use "hdfs find" to make a large search: # hdfs dfs -checksum `hdfs dfs -find /tmp -print` The above command will list checksum of all the files. You can also run with "sort and uniq " as : hdfs dfs -checksum `hdfs dfs -find /tmp -print` | sort | uniq -c | awk '{print $2,$4}' ... View more