Some additional things to consider: Cost of transporting data: Azure bills for network usage. This is not an issue, for example, if the MSSQL that you are ingesting data from is also in Azure. If data is going to live in the cluster for long, e.g. several weeks, then your best bang for buck is going to be to host it in your datacenter on bare metal. Obviously, an important argument in favor of HDInsight would be savings in terms of ease of managing the cluster. Also lack of in house speed, skill and ability to host a cluster in your DC would preclude this option. Why is that? Because it goes against the grain of a basic tenet of Hadoop: "take processing to data instead of taking processing to data". HDInsight does not store data data locally; it is stored in Azure Blob Storage. So all data must be brought to processing (from Azure cloud storage to computer nodes of the cluster). This is more important if you are doing I/O heavy processing, e.g. running data intensive MR loads like hive queries against data in DFS backed by Azure Blob Storage. For comparison, if you were running, say, a Spark load then this may not be an issue because the main bottleneck is compute not data transport. In general, HDInsight might be best suited for a targeted workload where you fire up a temporary cluster do your analysis and then take it down. For completeness, I should mention that HDInsight does have a tiny local DFS but that is to store temporary files created during MR runs.
... View more
