question Re: Recommendations for Microsoft Azure HDP Deployment in Support Questions

Recommendations for Microsoft Azure HDP Deployment

amcbarnett — Fri, 16 Sep 2022 09:45:36 GMT

Microsoft Azure Sizing Details (http://azure.microsoft.com/en-us/pricing/details/virtual-machines/#Linux)
1. You need to size and price machine and storage separately.
2. Do not use A8 machines. Use either A10 or A11’s. A8 is backed by Infiniband which is more expensive and unnecessary for Hadoop
3. Recommend D Series for Solid State Drives if needed.
4. Both options will need attached Blob Storage. The 382 GB local disk that comes with the VM is just for temp storage.
5. For Blob Storage, it comes in 1023GB sizes. Each VM has a maximum number of Blob Storage that can be attached. Eg. A10 Vms can have a maximum of 16 * 1TB storage. See the following for more details:
According to Microsoft it is recommended that Page Blob Storage is used opposed to Block Storage, (see http://azure.microsoft.com/en-us/pricing/details/storage/)
If performance is a must, especially with Kafka and Storm, use Premium storage not Standard.

UPDATED

- Converted into an article for more updated information. See

https://community.hortonworks.com/articles/22376/recommendations-for-microsoft-azure-hdp-deployment-1.html

Re: Recommendations for Microsoft Azure HDP Deployment

alal1 — Sat, 24 Oct 2015 05:32:03 GMT

Decision about using Page blob vs Block blob can be bit more nuanced, at least, when it comes to using Azure Blob store for HDFS. This page provides good overview: https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Page_Blob_Support_and_Configuration.

Re: Recommendations for Microsoft Azure HDP Deployment

dorio — Fri, 11 Dec 2015 06:06:44 GMT

I think it's also a good starting point to use Availability sets for master nodes and worker nodes. Another good point is about using one storage account for every nodes in the cluster in order to bypass IOPS limits for multiple VMs on the same Storage Account. You can also try to use Azure Data Lake Store (with adl://) in order to check the performance on the new Azure service.

You also need to remember the maintenance windows of every Azure region according to your customers: some regions could be a good choice for new service availability (e.g.: US East 2) but not from a maintenance point of view (expecially for european customers).

We also verified great differences between IaaS performance and PaaS (HDInsight) performance due to low read/write performance of the Blob Storage: with the first one (configured correctly) you can achieve best performance.

Re: Recommendations for Microsoft Azure HDP Deployment

pardeep_kumar — Tue, 15 Dec 2015 12:37:33 GMT

@Andrea D'Orio

Good point but I have a follow up question. How did you get past of storage account limit per subscription as it's limited to 100 Storage accounts per subscription? Did you get it increased from Microsoft or are you using multiple subscription for 100+ nodes cluster?

Re: Recommendations for Microsoft Azure HDP Deployment

dorio — Tue, 15 Dec 2015 15:28:57 GMT

@Pardeep Yes, we had to ask Microsoft Azure Support to increase limits for both Cores and Storage Account. Look at this link: https://azure.microsoft.com/en-us/blog/azure-limits-quotas-increase-requests/

Re: Recommendations for Microsoft Azure HDP Deployment

pardeep_kumar — Wed, 16 Dec 2015 07:21:35 GMT

DS14 with 10SSD disks would be what I will recommend for worker nodes. Blob storage would be good for cold backup but I don't see much value in using that for HDP workloads.

Re: Recommendations for Microsoft Azure HDP Deployment

wendy_foslien — Wed, 16 Dec 2015 20:22:42 GMT

@Andrea D'Orio do you have any benchmarking data on Azure IaaS vs. PaaS that you would be willing to share?

Re: Recommendations for Microsoft Azure HDP Deployment

sroberts — Tue, 12 Jan 2016 23:41:08 GMT

I've been told that the DV2 series is now recommended over the A series. Shall we update this? https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-size-specs/#standard-tier-dv2-series

Re: Recommendations for Microsoft Azure HDP Deployment

sroberts — Wed, 13 Jan 2016 02:53:22 GMT

Mind if we convert this to an Article and update together since no answer will be correct for more than a couple months?

Re: Recommendations for Microsoft Azure HDP Deployment

amcbarnett — Fri, 11 Mar 2016 01:34:46 GMT

@Sean Roberts

Good idea. Let me convert into an article