Next Steps after deploying Hortonworks Data Platform Standard on Azure

[#1]

I've received a couple of "how to" questions after folks successfully deploy the Hortonworks Data Platform Standard on Microsoft's Azure. I've collected my responses here as a reference to others:

What is Hortonworks Data Platform (HDP) Standard?

It is a muti-node HDP 2.4/Ambari 2.2.1 cluster on Microsoft’s Azure Cloud launched in a few mouse clicks. Hortonworks aims to match this service with the latest version of HDP.

You provide:

• your name
• email
• passwords or ssh key
• the number of nodes
• the VM types for your masters and workers
• should your cluster be HA or not

as shown in this screenshot:

Where's Ambari?

Once the cluster is successfully deployed, the Azure Dashboard will go from something like:

to

the Ambari service is located on the first master server. In the Azure portal goto Resource group, <your resource group name which was selected at first "Basics" step>, master1, settings and look for Public IP address. Use a web browser to access Ambari with:

<master1 Public IP address>:8080

What's the Ambari username?

The default username is "admin". The password was set under "Ambari password" in the screenshot above.

What are my ssh parameters?

The HDP service ports are enabled by default during the cluster installation. The master nodes allow external ssh access so use the cluster creation fields in the screenshot above in a terminal:

ssh <cluster admin username>@<your cluster name>-master-01.cloudapp.net

Worker nodes are only accessible via ssh from any of the master nodes.

Why am I receiving "Operation result in exceeding quota limits on Core"?

The default Azure Resource Manager (ARM) cores is 20 and not enough to deploy an HDP Standard cluster. Prior to deploying the cluster, request a ARM core quota increase to at least 120. Details to request a quota increase are here and remember ARM core resources are Azure region specific.

[#2]

Hi Ammet, I have found very useful your FAQ about to deploy an HDP cluster on Azure. However, I have some more questions:

1. Where can I see the HDP version in the Azure Marketplace before I deploy and pay for the cluster? I haven't found it, but you state that is the HDP version 2.4.
2. The are some restrictions about cluster size and virtual machines type. I need a cluster for a pre production stage so I am thinking of using 6 virtual machines azure D4_V2 (3 master and 3 slaves). Until I know, this would be the minimum cluster configuration for HDP on Azure. Later I could scale this cluster by adding / changing virtual machines. What do you think about this cluster configuration for pre production?

Thanks in advance,

Roberto Tardío

[#3]

Hi Roberto,

Glad the article was helpful! In reply to your questions:

1. Correct, the HDP version is not listed on the Azure Marketplace website. It's certainly something will consider. I believe we were trying to reduce the burden on the Microsoft site admins to constantly manage version/link/documentation links.

The HDP Azure Marketplace version should match the release cadence of HDP. And after you deploy HDP you can always check the HDP version in Ambari by going to Admin --> Stacks and Versions --> Version. The latest documentation and HDP releases notes are always here.

2. The Azure Marketplace deployment is great for non-elastic clusters and to start running a pilot use case. If ease of elasticity is a core requirement, take a look at Azure HDInsight to spin up more nodes on demand automatically. Not all the HDP services are on HDInsights but it may be a great option for your pilot.

Thanks,

Ameet