Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
New Contributor

How to read fsimage:

We can use Offline Image Viewer tool to view the fsimage data in a human readable format. Sometimes, this becomes more essential to analyse the fsimage to understand the usage pattern, how many 0 bite files are created, what is the space consumption pattern and is the fsimage corrupt.

Donwload the fsimage: hdfs dfsadmin –fetchImage /fsimage

This will download the latest fsimage from Name node

16/12/27 05:40:43 INFO namenode.TransferFsImage: Opening connection to http://<nn_hostname>:50070/getimage?getimage=1&txid=latest

16/12/27 05:40:43 INFO namenode.TransferFsImage: Transfer took 0.23s at 89.74 KB/s

Reading fsimage:

We can read the fsimage in several output formats.

1 .Web is the default output format.

2 .XML document

3 .Delimiters

4 .Reverse XML.

5 .FileDistribution is the tool for analyzing file sizes in the namespace image.

In this blog I will focusing on two output formats Web and Delimiters.

To get the output on web:

Run the oiv command with fsimage as input file:

hdfs oiv –i /fsimage/fsimage_0000000000000005792

16/12/27 05:48:43 INFO offlineImageViewer.FSImageHandler: Loading 9 strings

16/12/27 05:48:43 INFO offlineImageViewer.FSImageHandler: Loading 64 inodes.

16/12/27 05:48:43 INFO offlineImageViewer.FSImageHandler: Loading inode references

16/12/27 05:48:44 INFO offlineImageViewer.FSImageHandler: Loaded 0 inode references

16/12/27 05:48:44 INFO offlineImageViewer.FSImageHandler: Loading inode directory section

16/12/27 05:48:44 INFO offlineImageViewer.FSImageHandler: Loaded 32 directories

16/12/27 05:48:44 INFO offlineImageViewer.WebImageViewer: WebImageViewer started. Listening on /127.0.0.1:5978. Press Ctrl+C to stop the viewer.

Now open another terminal and run the below commands to read fsimage.

hdfs dfs -ls webhdfs://127.0.0.1:5978/

OR

hdfs dfs -ls –R webhdfs://127.0.0.1:5978/

10828-1.png

We can also get the output in JSON format by using curl: curl -i http://127.0.0.1:5978/webhdfs/v1/?op=liststatus

10829-2.png

To get the output in to an output directory: hdfs ovi –p Delimited –i /fsimage/fsimage__0000000000000005792 –o /fsimage/fsimage.txt

10830-3.png

We can read the data in fsimage.txt by running: head fsimage.txt from the local folder.

10841-4.png

4,007 Views
Comments
Not applicable

I'm getting OOM.

I have tried:

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx20G "; hdfs oiv ..
export HADOOP_OPTS="$HADOOP_OPTS -Xmx20G "; hdfs oiv ...
export JAVA_HEAP_MAX="-Xmx10G"; hdfs oiv ...

None worked.

Can anybody show how to do it?

Thx

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 06:37 AM
Updated by:
 
Contributors
Top Kudoed Authors