question Re: HDFS file and block size in Archives of Support Questions (Read Only)

HDFS file and block size

naveen1 — Fri, 16 Sep 2022 09:57:40 GMT

I got below details through hadoop fsck /

Total size: 41514639144544 B (Total open files size: 581 B)

Total dirs: 40524

Total files: 124348 Total symlinks: 0 (Files currently being written: 7)

Total blocks (validated): 340802 (avg. block size 121814540 B) (Total open file blocks (not validated): 7) Minimally replicated blocks: 340802 (100.0 %)

I am usign 256MB block size. so 340802 blocks * 256 MB = 83.2TB * 3(replicas) =249.6 TB but in cloudera manager it shows 110 TB disk used. how is it possible?

Does this mean even though block size is 256MB, small file doesnt use the whole block for itself?

Re: HDFS file and block size

Harsh J — Fri, 15 Jan 2016 18:51:58 GMT

Yes, blocks are not pre-allocated. They are a logical division unit. Read
https://wiki.apache.org/hadoop/FAQ#If_a_block_size_of_64MB_is_used_and_a_file_is_written_that_uses_less_than_64MB.2C_will_64MB_of_disk_space_be_consumed.3F

Re: HDFS file and block size

naveen1 — Fri, 15 Jan 2016 19:19:48 GMT

I have 194945 files that are less than 50MB and these files occupying 884GB memory. how to calculate the memory that these files will occupy if I hadoop archive them. 2) Am I using my hdfs efficiently as there are small files and I am not wasting any memory here. 3) Does archiving really save my disk space or it just reduces the namesapce ovevrhead. Harsh can you give me a detailed picture of this.

Re: HDFS file and block size

naveen1 — Fri, 15 Jan 2016 19:27:41 GMT

So even though I archive these files, I wont be saving any disk space, is that right.

Re: HDFS file and block size

ben.hemphill — Mon, 18 Jan 2016 22:10:50 GMT

You won't save HDFS filesystem space by "archiving" or "combining" small files. In many scenarios you will get a performance boost from combining. You will also reduce the metadata overhead on the namenode by combining as well.