consequences of changing block size?

[#1]

Hi,

i copied file1 of 512MB to hdfs at that time the block size is 128mb replication factor is 3.

after some time i have changed block size to 64mb using dfs.blocksize.

again i have copied one more file i.e file2 which is again 512 mb

now the file2 each block size is 64mb.

now my question when i changed block size 128mb to 64mb what will be the file1 block size?

does it change to 128mb to 64mb or it will continue as 128mb only?

please help me out

Thanks in advance

[#2 · Accepted solution]

Hi @pavan p

When you changing the block size from one value to other then only the files which are ingested/created in HDFS will be created with new block size. Where as the old files will remain to exists in the previous block size only and it will not changed. If you need to change then manual intervention is needed. Hope it Helps!

[#3]

@pavan p If it answers you question please choose it as a best answer!

[#4]

Hi @pavan p,

The file1 will remain to have block size as 128MB only.

If you want to change the block size of file1 to 64MB , you can use Hadoop Copy command as below , which will create a copy of file1 to file3 with block size mentioned in dfs.blocksize property.

$ hadoop fs -cp /user/ubuntu/file1 /user/ubuntu/file3

Also you can specify the blocksize parameter in the command as below:

$ hadoop fs -D dfs.blocksize=xx -cp /user/ubuntu/file1 /user/ubuntu/file3

The only thing you need to do is , manually delete the old copy of the file if it is no longer needed !