@Viraj Vekaria Yes. HDFS does the automated sharding and you have no control on it, but rarely one thinks about sharding of a file system like HDFS, but of an actual database like RDBMS, MongoDB or HBase. Semantics, but you asked to use the sharding in HDFS which implies manual control. It is done automatically. At most, what you could do is to change the global replication factor, change the replication factor per file, but you can't do anything about what is replicated where, no data locality control. Since @Justin Watkins mentioned traditional RDBMS and I mentioned also MongoDB and you asked about HDFS, I will summarize the differences in approach to achieve scalability between these three with the added HBase touch. Traditional RDBMS often run into bottlenecks with scalability and data replication when handling large amounts of data/data sets. There are some creative ways to setup master-slave setups to achieve some scalability and performance and all are coming by design, not out of box sharding. MongoDB sharding can be applied to allow distribution across multiple systems for horizontal scalability as needed. Like MongoDB, Hadoop’s HBase database accomplishes horizontal scalability through database sharding. Distribution of data storage is handled by the HDFS, with an optional data structure implemented with HBase, which allocates data into columns (versus the two-dimensional allocation of an RDBMS in columns and rows). Data can then be indexed (through use of software like Solr), queried with Hive, or have various analytics or batch jobs run on it with choices available from the Hadoop ecosystem or your choice of business intelligence platform. If any of the responses is helpful, please don't forget to vote and accept the best answer. Thanks.
... View more
