Recommendations for Microsoft Azure HDP Deployment

[#1]

• Microsoft Azure Sizing Details (http://azure.microsoft.com/en-us/pricing/details/virtual-machines/#Linux)
  1. You need to size and price machine and storage separately.
  2. Do not use A8 machines. Use either A10 or A11’s. A8 is backed by Infiniband which is more expensive and unnecessary for Hadoop
  3. Recommend D Series for Solid State Drives if needed.
  4. Both options will need attached Blob Storage. The 382 GB local disk that comes with the VM is just for temp storage.
  5. For Blob Storage, it comes in 1023GB sizes. Each VM has a maximum number of Blob Storage that can be attached. Eg. A10 Vms can have a maximum of 16 * 1TB storage. See the following for more details:
  1. https://msdn.microsoft.com/library/azure/dn197896.aspx
  2. https://azure.microsoft.com/en-us/documentation/articles/cloud-services-sizes-specs/
  3. https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-size-specs/
• According to Microsoft it is recommended that Page Blob Storage is used opposed to Block Storage, (see http://azure.microsoft.com/en-us/pricing/details/storage/)
• If performance is a must, especially with Kafka and Storm, use Premium storage not Standard.

UPDATED

- Converted into an article for more updated information. See

https://community.hortonworks.com/articles/22376/recommendations-for-microsoft-azure-hdp-deployment-...

[#2]

Decision about using Page blob vs Block blob can be bit more nuanced, at least, when it comes to using Azure Blob store for HDFS. This page provides good overview: https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Page_Blob_Support_and_Configuration.

[#3 · Accepted solution]

I think it's also a good starting point to use Availability sets for master nodes and worker nodes. Another good point is about using one storage account for every nodes in the cluster in order to bypass IOPS limits for multiple VMs on the same Storage Account. You can also try to use Azure Data Lake Store (with adl://) in order to check the performance on the new Azure service.

You also need to remember the maintenance windows of every Azure region according to your customers: some regions could be a good choice for new service availability (e.g.: US East 2) but not from a maintenance point of view (expecially for european customers).

We also verified great differences between IaaS performance and PaaS (HDInsight) performance due to low read/write performance of the Blob Storage: with the first one (configured correctly) you can achieve best performance.

[#4]

@Andrea D'Orio

Good point but I have a follow up question. How did you get past of storage account limit per subscription as it's limited to 100 Storage accounts per subscription? Did you get it increased from Microsoft or are you using multiple subscription for 100+ nodes cluster?

[#5]

@Pardeep Yes, we had to ask Microsoft Azure Support to increase limits for both Cores and Storage Account. Look at this link: https://azure.microsoft.com/en-us/blog/azure-limits-quotas-increase-requests/

[#6]

@Andrea D'Orio do you have any benchmarking data on Azure IaaS vs. PaaS that you would be willing to share?

[#7]

DS14 with 10SSD disks would be what I will recommend for worker nodes. Blob storage would be good for cold backup but I don't see much value in using that for HDP workloads.

[#8]

I've been told that the DV2 series is now recommended over the A series. Shall we update this? https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-size-specs/#standard-tier-...

[#9]

Mind if we convert this to an Article and update together since no answer will be correct for more than a couple months?

[#10]

@Sean Roberts

Good idea. Let me convert into an article