List Of Block Placement Algorithm For HDFS

[#1]

With HDFS-385, pluggable interface for HDFS block's replica placement is allowed. I found multiple HDFS JIRAs pointing to various such algorithm, yet didn't find any documentation listing the available choices. JIRA # HDFS-8789 will allow migration of block placement policy, yet the JIRA has no further details. Any help in identifying the list of such algorithms.

[#2 · Accepted solution]

A quick search in the code base tells me that we have these following policies

• AvailableSpaceBlockPlacementPolicy
• BlockPlacementPolicyDefault
• BlockPlacementPolicyRackFaultTolerant
• BlockPlacementPolicyWithNodeGroup
• BlockPlacementPolicyWithUpgradeDomain

> yet didn't find any documentation listing the available choices.

You are absolutely right, we can certainly do better on documenting this, Thanks for pointing this out. I will address this in an Apache JIRA.

[#3]

@smdas Sorry forgot to tag you.

[#4]

@aengineer In continuation of Smarak's concern, do we have any further info on this?

JIRA HDFS-8789 is still in unresolved state, so looks change in block placement policy is still not applicable to existing blocks.

Any thoughts?

[#5]

@aengineer 

HDFS-14637 and HDFS-8789 appear to contradict each other as HDFS-14637 says that after changing the network topology or placement policy on a cluster and restarting the namenode, the namenode will scan all blocks on the cluster at startup, and check if they meet the current placement policy. If they do not, they are added to the replication queue and the namenode will arrange for them to be replicated to ensure the placement policy is used. 

It would be good to get some clarity on this.  

[#6]

@aengineer 

Also noticed from one comment in HDFS-8789 that "balancer doesn't support anything other than the default placement policy (BlockPlacementPolicyDefault)."

HDFS-14053 says ability for NN to re-replicate blocks based on policy change is fixed in hadoop 3.3.0 [not sure if it's hadoop version or not, though NN version doesn't make sense], while HDFS-14637 supports above statement until UD get enable.