Support Questions
Find answers, ask questions, and share your expertise

HDFS BlockPlacementPolicy, is there an alternative that considers available disk space?

Solved Go to solution
Highlighted

HDFS BlockPlacementPolicy, is there an alternative that considers available disk space?

Contributor

Hello,

I am wondering if there is an BlockPlacementPolicy that in addition to storing replicas safely on different racks as the default one does, also can consider how much disk space that is available on different nodes?

In case where you have a cluster that consists of two sets of machines with a big difference in the amount of available disk space, the default policy will lead to the disks of the set with a smaller amount of disk space running out of disk space long before you actually reach your total HDFS capacity.

Is there any such policy ready to be used?

Best Regards

Thomas

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HDFS BlockPlacementPolicy, is there an alternative that considers available disk space?

Contributor

I just found that something like this was added somewhat recently:

https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/h...

This seems to be what I was looking for.

View solution in original post

1 REPLY 1

Re: HDFS BlockPlacementPolicy, is there an alternative that considers available disk space?

Contributor

I just found that something like this was added somewhat recently:

https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/h...

This seems to be what I was looking for.

View solution in original post