@Viraj Vekaria Yes. HDFS does the automated sharding and you have no control on it, but rarely one thinks about sharding of a file system like HDFS, but of an actual database like RDBMS, MongoDB or HBase. Semantics, but you asked to use the sharding in HDFS which implies manual control. It is done automatically. At most, what you could do is to change the global replication factor, change the replication factor per file, but you can't do anything about what is replicated where, no data locality control. Since @Justin Watkins mentioned traditional RDBMS and I mentioned also MongoDB and you asked about HDFS, I will summarize the differences in approach to achieve scalability between these three with the added HBase touch. Traditional RDBMS often run into bottlenecks with scalability and data replication when handling large amounts of data/data sets. There are some creative ways to setup master-slave setups to achieve some scalability and performance and all are coming by design, not out of box sharding. MongoDB sharding can be applied to allow distribution across multiple systems for horizontal scalability as needed. Like MongoDB, Hadoop’s HBase database accomplishes horizontal scalability through database sharding. Distribution of data storage is handled by the HDFS, with an optional data structure implemented with HBase, which allocates data into columns (versus the two-dimensional allocation of an RDBMS in columns and rows). Data can then be indexed (through use of software like Solr), queried with Hive, or have various analytics or batch jobs run on it with choices available from the Hadoop ecosystem or your choice of business intelligence platform. If any of the responses is helpful, please don't forget to vote and accept the best answer. Thanks. ... View more

@Raja Sekhar Chintalapati If you want your password to be encrypted you need to have SSL enabled and use -ssl=true flag in the connect string. ++++ If a response was helpful to address your matter, please vote and accept the best answer. If you have a better answer, please add it and a moderator will review it and accept it as it stands correct. Thanks. ... View more

@Sandeep Nemuri Without knowing the problem you are trying to resolve, there is no fair comparison between two architectures that are meant to address different functionality. It depends on the type of problems you are trying to resolve. Sorry to add this caveat and not to provide a silver bullet. #1: if you have massive data stored and you need the distributed power of Spark to fetch the data and generate the data frames which are then consumed by the R application, then the following architecture is recommended. R client -> R Server -> SparkR -> Spark -> Data Source (usually Hadoop) #2: if you are building Spark applications that eventually need access to some R functions that are delivered by the R server then your architecture looks more like this: Spark client -> Spark on YARN -> SparkR -> R Server The first case would be what you call R on Spark. The second case would be what you call Spark on R. My observation on multiple customers is that Spark applications use #2 and R applications use #1. A data scientist that uses R as his/her tool and only needs Spark to handle the massive data back and forth, will use #1. A data scientist or an application developer that needs to deliver a Spark application and wants to leverage some existent R functionality will use #2. It is obvious that due to its distributed nature and taking advantage of resources from multiple nodes, an architecture like #1 will benefit an R application that needs Spark's muscle in the cluster, while #2 already has the Spark muscle and brain and will need also some of the R brain. Your R server has lots of brain but not a lot of muscle when is to compare it with Spark on YARN over a large cluster with lots of CPU and RAM. If a response helped to shed some light please don't forget to vote or accept the best answer. If you have a better answer, please add it and a moderator will review it and eventually accept it. ... View more