question Re: Disk size used is bigger than replication number multiplied by files size in Support Questions

Disk size used is bigger than replication number multiplied by files size

dvt_isoft — Sun, 18 Aug 2019 11:23:35 GMT

Hi,

I am running Hadoop on a 3 nodes cluster (3 virtual machines) with respectively 20Gb, 10Gb and 10Gb of disk space available.

When I run this command on the namenode :

hadoop fs -df -h /

I get the following result :

When I run this command :

hadoop fs -du -s -h /

I get the following result :

Knowing that the replication number is set to 3, shouldn't I get 3*2,7 = 8,1G in the first screenshot ?

I tried to execute expunge command and it did not change the result.

Thanks in advance !

Sylvain.

Re: Disk size used is bigger than replication number multiplied by files size

bhavintandel — Mon, 20 Mar 2017 19:05:57 GMT

Can you please check if the screenshots are uploaded properly because it is not seen on this end.

Re: Disk size used is bigger than replication number multiplied by files size

dvt_isoft — Mon, 20 Mar 2017 20:04:24 GMT

It should be alright now.

Re: Disk size used is bigger than replication number multiplied by files size

cstanca — Tue, 21 Mar 2017 01:00:30 GMT

@dvt isoft

Not necessarily. That would be only if your blocks will be 100% filled with data.

Let's say you have a 1024 MB file and the block size is 128 MB. That would be exactly 8 blocks at 100%.

Let's say you have 968 MB file and the block size is128 MB. That is still 8 blocks but with lower usage. A block once used by a file cannot be reused for a different file.

That's why loading small files could be a waste.

Just imagine 100 files of each 100 KB will be using 100 blocks for 128 MB, 10x more than the examples I provided above.

You need to understand your files, block % usage etc.

The command you execute shows the blocks empty x size/block ... I know that is confusing 🙂

+++

If this is helpful please vote and accept as the best answer.