How to read fsimage: We can use Offline Image Viewer tool to view the fsimage data in a human readable format. Sometimes, this becomes more essential to analyse the fsimage to understand the usage pattern, how many 0 bite files are created, what is the space consumption pattern and is the fsimage corrupt.
Donwload the fsimage:
hdfs dfsadmin –fetchImage /fsimage This will download the latest fsimage from Name node 16/12/27 05:40:43 INFO namenode.TransferFsImage: Opening connection to http://<nn_hostname>:50070/getimage?getimage=1&txid=latest 16/12/27 05:40:43 INFO namenode.TransferFsImage: Transfer took 0.23s at 89.74 KB/s
Reading fsimage: We can read the fsimage in several output formats. 1 .Web is the default output format. 2 .XML document
3 .Delimiters
4 .Reverse XML. 5 .FileDistribution is the tool for analyzing file sizes in the namespace image. In this blog I will focusing on two output formats Web and Delimiters. To get the output on web:
Run the oiv command with fsimage as input file: hdfs oiv –i /fsimage/fsimage_0000000000000005792 16/12/27 05:48:43 INFO offlineImageViewer.FSImageHandler: Loading 9 strings 16/12/27 05:48:43 INFO offlineImageViewer.FSImageHandler: Loading 64 inodes. 16/12/27 05:48:43 INFO offlineImageViewer.FSImageHandler: Loading inode references 16/12/27 05:48:44 INFO offlineImageViewer.FSImageHandler: Loaded 0 inode references 16/12/27 05:48:44 INFO offlineImageViewer.FSImageHandler: Loading inode directory section 16/12/27 05:48:44 INFO offlineImageViewer.FSImageHandler: Loaded 32 directories 16/12/27 05:48:44 INFO offlineImageViewer.WebImageViewer: WebImageViewer started. Listening on /127.0.0.1:5978. Press Ctrl+C to stop the viewer. Now open another terminal and run the below commands to read fsimage.
hdfs dfs -ls webhdfs://127.0.0.1:5978/ OR hdfs dfs -ls –R webhdfs://127.0.0.1:5978/ We can also get the output in JSON format by using curl:
curl -i http://127.0.0.1:5978/webhdfs/v1/?op=liststatus To get the output in to an output directory:
hdfs ovi –p Delimited –i /fsimage/fsimage__0000000000000005792 –o /fsimage/fsimage.txt We can read the data in fsimage.txt by running: head fsimage.txt from the local folder.
... View more
