question Re: List Of Block Placement Algorithm For HDFS in Support Questions

List Of Block Placement Algorithm For HDFS

smdas — Fri, 09 Mar 2018 20:11:58 GMT

With HDFS-385, pluggable interface for HDFS block's replica placement is allowed. I found multiple HDFS JIRAs pointing to various such algorithm, yet didn't find any documentation listing the available choices. JIRA # HDFS-8789 will allow migration of block placement policy, yet the JIRA has no further details. Any help in identifying the list of such algorithms.

Re: List Of Block Placement Algorithm For HDFS

aengineer — Sat, 10 Mar 2018 02:33:18 GMT

A quick search in the code base tells me that we have these following policies

AvailableSpaceBlockPlacementPolicy
BlockPlacementPolicyDefault
BlockPlacementPolicyRackFaultTolerant
BlockPlacementPolicyWithNodeGroup
BlockPlacementPolicyWithUpgradeDomain

> yet didn't find any documentation listing the available choices.

You are absolutely right, we can certainly do better on documenting this, Thanks for pointing this out. I will address this in an Apache JIRA.

Re: List Of Block Placement Algorithm For HDFS

aengineer — Sat, 10 Mar 2018 02:34:03 GMT

@smdas Sorry forgot to tag you.

Re: List Of Block Placement Algorithm For HDFS

aps — Mon, 06 Jan 2020 11:09:10 GMT

@aengineer In continuation of Smarak's concern, do we have any further info on this?

JIRA HDFS-8789 is still in unresolved state, so looks change in block placement policy is still not applicable to existing blocks.

Any thoughts?

Re: List Of Block Placement Algorithm For HDFS

aps — Mon, 06 Jan 2020 16:08:29 GMT

@aengineer

HDFS-14637 and HDFS-8789 appear to contradict each other as HDFS-14637 says that after changing the network topology or placement policy on a cluster and restarting the namenode, the namenode will scan all blocks on the cluster at startup, and check if they meet the current placement policy. If they do not, they are added to the replication queue and the namenode will arrange for them to be replicated to ensure the placement policy is used.

It would be good to get some clarity on this.

Re: List Of Block Placement Algorithm For HDFS

aps — Mon, 06 Jan 2020 16:57:30 GMT

@aengineer

Also noticed from one comment in HDFS-8789 that "balancer doesn't support anything other than the default placement policy (BlockPlacementPolicyDefault)."

HDFS-14053 says ability for NN to re-replicate blocks based on policy change is fixed in hadoop 3.3.0 [not sure if it's hadoop version or not, though NN version doesn't make sense], while HDFS-14637 supports above statement until UD get enable.