Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How hdfs generates datanode path?

avatar
Explorer

Reference : https://stackoverflow.com/questions/51930336/how-datanode-path-is-created-in-hadoop

How do hdfs generates datanode path? Say for example I'm asking hdfs to write a file in hdfs location hdfs://192.168.143.150:9000/filestore/TAO.mp4 it is getting written in the path "/data/hadoop-data/dn/current/BP-1308070615-172.22.131.23-1533215887051/current/finalized/subdir0/subdir0/blk_1073741869".

How this path gets generated?

1 ACCEPTED SOLUTION

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Contributor

@rabbit,

the path is already defined on Ambari (hdfs -> configs -> DataNode Directories) this is the path you defined for HDFS to write the block information on the actual disk location on each data node.

in your case: This path must have defined on ambari as: /data/hadoop-data/dn/ - under this, HDFS creates remaining folders starting from "current"

Please check your ambari -> hdfs properties and confirm.

I hope this help you.

avatar
Explorer

Thanks @hdpadmin. You were right I have configured "/data/hadoop-data/dn" in hadoop-site.xml. But why does hdfs create multiple subfolders within the current folder like "BP-1308070615-172.22.131.23-1533215887051/current/finalized/subdir0/subdir0"? What does the sub directory name mean?

avatar

Hi @rabbit s

The BP stands for "block pool", a collection of blocks belonging to a single HDFS namespace.

The next part 1308070615, is a randomly generated integer.
The IP address is the address of the NameNode that originally created the block pool
The last part is the creation time of the namespace.

You can read more about this here;

https://hortonworks.com/blog/hdfs-metadata-directories-explained/

avatar
Explorer

This is exactly what I was looking for ! Thanks a lot @Jonathan Sneep

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Why does this again creating "current/finalized/subdir0/subdir0" ?