question Re: File distribution on HDFS in Archives of Support Questions (Read Only)

File distribution on HDFS

fil — Mon, 09 Nov 2015 03:40:37 GMT

Hi dear expert!

i'm wondering is there any way to check file distribution amond nodes in HDFS?

some way that allow to check on which nodes place some particular file of dirrectory?

thanks!

Re: File distribution on HDFS

Harsh J — Mon, 09 Nov 2015 04:03:56 GMT

You can check the file on NN Web UI's File Browser (shows all block IDs and
their locations), or run 'hdfs fsck /path/to/file -files -blocks -locations'

Re: File distribution on HDFS

fil — Mon, 09 Nov 2015 04:05:56 GMT

Thanks! Hdfs fsck will works, but it's hard to analyze in case of big file. maybe it is other way to get aggregate values?

Re: File distribution on HDFS

Harsh J — Mon, 09 Nov 2015 07:51:56 GMT

If by 'hard to analyse' you mean to parse/process it, you can consider
using the Java API to fetch block location info too:
http://archive.cloudera.com/cdh5/cdh/5/hadoop/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,%20long,%20long)