@Saravana V To change the block size, parameter, dfs.block.size can be changed to required value(default in hadoop 2.0 is 128mb 256mb in hdfs-site.xml file. Once this is changed through Ambari UI the ONLY recommended way, the cluster restart is required for the change to effect, for which will be applied only to the new files. Change this setting and restart all stale configurations see 256.JPG desktop --Created a directory for the test if it does not exist Copied a file specifying the block size to 256 MB see new_file256.JPG $ hdfs dfs -D dfs.blocksize=268435456 -put /tmp/ambari.properties.4 /user/sheltong/test Copied the new files to the same directory of files size 128 see new2_file256.JPG $ hdfs dfs -D dfs.blocksize=268435456 -put /tmp/ambari.properties.4 /user/sheltong DISTCP see (distributed copy) is a tool used for large inter/intra-cluster copying. It uses MapReduce to effect its distribution, error handling and recovery and reporting. To overwrite the old 128MB files copy the new files with the new block size to the new location adding the option overwrite. However, we have to manually delete the old files with the older block size. Command: $ hadoop distcp -Ddfs.block.size=268435456 /path/to/data(Source) /path/to/data-with-largeblocks(Destination). Now the question becomes should I make my dataset 128 MB or 256 MB or even more? It all depends on your cluster capacity and the size of your datasets. Let's say you have a dataset which is 2 Petabytes in size. Having a 64 MB block size for this dataset will result in 31 million+ blocks which would put stress on the NameNode to manage all that blocks. Having a lot of blocks will also result in a lot of mappers during MapReduce execution. So, in this case, you may decide to increase the block size just for that dataset. Hope that helps
... View more
