Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
File distribution on HDFS
Labels:
- Labels:
-
HDFS
Rising Star
Created 11-08-2015 07:40 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi dear expert!
i'm wondering is there any way to check file distribution amond nodes in HDFS?
some way that allow to check on which nodes place some particular file of dirrectory?
thanks!
1 ACCEPTED SOLUTION
Mentor
Created 11-08-2015 11:51 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If by 'hard to analyse' you mean to parse/process it, you can consider
using the Java API to fetch block location info too:
http://archive.cloudera.com/cdh5/cdh/5/hadoop/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockL...
using the Java API to fetch block location info too:
http://archive.cloudera.com/cdh5/cdh/5/hadoop/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockL...
3 REPLIES 3
Mentor
Created 11-08-2015 08:03 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can check the file on NN Web UI's File Browser (shows all block IDs and
their locations), or run 'hdfs fsck /path/to/file -files -blocks -locations'
their locations), or run 'hdfs fsck /path/to/file -files -blocks -locations'
Rising Star
Created 11-08-2015 08:05 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks! Hdfs fsck will works, but it's hard to analyze in case of big file. maybe it is other way to get aggregate values?
Mentor
Created 11-08-2015 11:51 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If by 'hard to analyse' you mean to parse/process it, you can consider
using the Java API to fetch block location info too:
http://archive.cloudera.com/cdh5/cdh/5/hadoop/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockL...
using the Java API to fetch block location info too:
http://archive.cloudera.com/cdh5/cdh/5/hadoop/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockL...