Support Questions
Find answers, ask questions, and share your expertise

How can I change the size of datanode(not block) in HDFS?

How can I change the size of datanode(not block) in HDFS?

I want to test the practical scenario that what if one of datanode is full in HDFS. For that I want to reduce the size of datanode so that I can fill it quickly.

Right now HDFS shows me the size of 96 GB. So thinking to make it 6 to 7 GB temporarily.

6 REPLIES 6

Re: How can I change the size of datanode(not block) in HDFS?

Contributor

Hi Viraj,

I guess you can set a quota limit using the command:

hadoop dfsadmin -setSpaceQuota <max_size> <directory>

For more information you can refer:

https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Space...

Regards, Karthik Gopal

Re: How can I change the size of datanode(not block) in HDFS?

Thanks Karthik,

I thought there might be some property exist in HDFS which will do this.Anyway I'll try this way.

Re: How can I change the size of datanode(not block) in HDFS?

This applies a quota to a directory in HDFS. It does not enforce anything at the layer of the DataNode's interaction with the local file system, so it cannot be used to simulate the scenario of a full DataNode.

Re: How can I change the size of datanode(not block) in HDFS?

Or just fill up the file system with some dummy non HDFS data. HDFS monitors free space on the datanode

Re: How can I change the size of datanode(not block) in HDFS?

Hello @Viraj Vekaria. I have 2 ideas for you:

  1. If you have flexibility to resize volumes, such as via LVM, then change to a smaller size at the OS layer, restart the DataNode, and then fill it up with data consume that small amount of remaining space.
  2. If you don't have flexibility to resize volumes, then you can limit the size of the volume considered for usage by HDFS by setting property dfs.datanode.du.reserved in hdfs-site.xml.
<property>
  <name>dfs.datanode.du.reserved</name>
  <value>0</value>
  <description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.
  </description>
</property>

Re: How can I change the size of datanode(not block) in HDFS?

Thanks @Chris Nauroth,I think 2nd option is better as I don't have that much feasibility to do 1st option.