question Re: How hdfs generates datanode path? in Support Questions

How hdfs generates datanode path?

rajasuba_s — Mon, 20 Aug 2018 19:59:50 GMT

Reference : https://stackoverflow.com/questions/51930336/how-datanode-path-is-created-in-hadoop

How do hdfs generates datanode path? Say for example I'm asking hdfs to write a file in hdfs location hdfs://192.168.143.150:9000/filestore/TAO.mp4 it is getting written in the path "/data/hadoop-data/dn/current/BP-1308070615-172.22.131.23-1533215887051/current/finalized/subdir0/subdir0/blk_1073741869".

How this path gets generated?

Re: How hdfs generates datanode path?

ssanupindi — Mon, 20 Aug 2018 21:57:37 GMT

@rabbit,

the path is already defined on Ambari (hdfs -> configs -> DataNode Directories) this is the path you defined for HDFS to write the block information on the actual disk location on each data node.

in your case: This path must have defined on ambari as: /data/hadoop-data/dn/ - under this, HDFS creates remaining folders starting from "current"

Please check your ambari -> hdfs properties and confirm.

I hope this help you.

Re: How hdfs generates datanode path?

rajasuba_s — Tue, 21 Aug 2018 16:28:56 GMT

Thanks @hdpadmin. You were right I have configured "/data/hadoop-data/dn" in hadoop-site.xml. But why does hdfs create multiple subfolders within the current folder like "BP-1308070615-172.22.131.23-1533215887051/current/finalized/subdir0/subdir0"? What does the sub directory name mean?

Re: How hdfs generates datanode path?

JonathanSneep — Tue, 21 Aug 2018 16:41:21 GMT

Hi @rabbit s

The BP stands for "block pool", a collection of blocks belonging to a single HDFS namespace.

The next part 1308070615, is a randomly generated integer.
The IP address is the address of the NameNode that originally created the block pool
The last part is the creation time of the namespace.

You can read more about this here;

https://hortonworks.com/blog/hdfs-metadata-directories-explained/

Re: How hdfs generates datanode path?

rajasuba_s — Tue, 21 Aug 2018 17:22:54 GMT

This is exactly what I was looking for ! Thanks a lot @Jonathan Sneep

Re: How hdfs generates datanode path?

rajasuba_s — Tue, 21 Aug 2018 17:24:31 GMT

Combining @hdpadmin 's answer and @Jonathan Sneep's answer gives me the best answer to my question 🙂 Thank you both 🙂

Re: How hdfs generates datanode path?

rajasuba_s — Tue, 21 Aug 2018 17:26:21 GMT

Why does this again creating "current/finalized/subdir0/subdir0" ?