@Christopher Amatulli Hadoop was created to work with local attached storage. The whole idea of bringing compute to storage. This enables you to have failure redundancy, parallel processing on local data and reliability as a disk or node failure will simply kick off an automatic mechanism to re replicate lost data. For most performance you should have your data local to where your compute is and where your job is running. So Spark should read data on a local partition, rather than a remote storage. That bing said, for cost purpose companies might put old data in low cost storage like S3 and then run their jobs on that remote storage. This works but with the expectation that it is going to be slow compared to reading data from local disk. So, it depends on what your requirements are. Are you going to have a PB of data? That might be a good reason to have remote low cost storage like S3 to save money. Depending on your requirements, you may keep storage separate but it is not how you would usually go about. Also, Hive doesn't need to be separate. It runs on your compute nodes. Hive reads data that is in your HDFS. It is not a database in the sense that it will have its own storage. So you store data in HDFS (whether local or remote storage) and create Hive tables on that data and run your queries. As you can imagine, it would be more efficient if the data is local. Hope this helps. Please feel free to comment if you have additional questions.
... View more
