Are you sure? I was pretty sure that even big files have their first copy written locally. ... View more

Normally they are not. Hadoop tries to write the first copy of every block locally if possible ( this has a lot of nice characteristics if the same nodes normally read the files they write ( for example HBase Region Servers. ) So the general rule is: One client writes a big file with 10 blocks: - HDFS tries to write a first version of each block to the local datanode if the uploading client is colocated with a datanode in rack1 ( otherwise any node is used) - While the block is written the datanode initiates a copy to another node in a different rack ( rack2 ) - while that block is written the second datanode initiates a copy to another node in the same rack as itself ( rack2 ) So you essentially have a copy chain, however not all three writes need to finish successfully. If at least the first copy is written successfully the block is assumed as written and the client will start writing the second block and the third and so on. But the first copy of each will per default end up on the same datanode colocated with the client. As said this has a lot of good results in practice. However all of that goes out the window when the datanode should run full. ... View more

I don't think there is any max number Most likely some theoretical int number that you wouldn't reach in any cluster. Or do you mean at the same time? In that case it would be (RAM * Nodenumber) / *yarn.scheduler.minimum-allocation-mb So if you have a 100 nodes and your nodes have 96GB yarn memory and your minimum allocation size ( and your map size ) is 2GB it would be 4800. ... View more

1) You can enable some parameters like huge file support in Linux to make it faster but nothing near 128MB. You just don;t get much benefit there If you want to know how to tune the filesystem you can refer to our reference architecture Appendix B https://hortonworks.com/wp-content/uploads/2013/10/4AA5-9017ENW.pdf - Mounting filesystems with nodiratime, noatime - Change block readahead to 8k Some file systems can also configure huge file support. 2) Windows Basically yes. NTFS has something they call a cluster. But it seems to be very similar to a block https://support.microsoft.com/en-gb/kb/140365 ... View more

I think that question was well answered in this thread. https://community.hortonworks.com/questions/27567/write-performance-in-hdfs.html#answer-27633 If I can quote myself, here the explanation of why Hadoop block size is much larger than a local filesystem block "2. As HDFS is a Virtual Filesystem, the data it stores will ultimately stored on Underlying Operating system (Most cases, Linux). Assuming Linux has Ext3 File system (whose block size is 4KB), How does having 64MB/128 MB Block size in HDFS help ? Does the 64 MB Block in HDFS will be split into x * 4 KB Blocks of underlying Opertating system ?" Again small additional comment to what Kuldeep said. A block is just a Linux file so yes all the underlying details of ext3 or whatever still apply. The reason blocks are so big is not because of the storage on the local node but because of the central FS storage - To have a distributed filesystem you need to have a central FS Image that keeps track of ALL the blocks on ALL the servers in the system. In HDFS this is the Namenode. He keeps track of all the files in HDFS and all the blocks on all the datanodes. This needs to be in memory to support the high number of concurrent tasks and operations happening in an hadoop cluster so the traditional ways of setting this up ( btrees ) doesn't really work. It also needs to cover all nodes on all discs so he needs to keep track of thousands of nodes with tens of thousands of drives. Doing that with 4kb blocks wouldn't scale so the block size is around 128MB on most systems. ( You can count roughly 1GB of Namenode memory for 100TB of data ) - If you want to process a huge file in HDFS you need to run a parallel task on it ( MapReduce, Tez, Spark , ... ) In this case each task gets one block of data and reads it. It might be local or not. Reading a big 128 MB block or sending him over the network is efficient. Doing the same with 30000 4KB files would be very inefficient. That is the reason for the block size. ... View more