How to set up secure HDP production cluster on Azure?

[#1]

Hi All,

We are looking forward to setup HDP on Azure to start with 18 node cluster. Mostly with A-8 to A-11 type instances.

I am looking for setup instruction and best practices.

I went through the following documents but I see they are a bit out dated and most of them tackle with sandbox.

Azure Sandbox - Hortonworks Blog

Deploying Sandbox in Azure - Hortonworks Tutorial

Azure Using Cloud break (We are not looking forward to use CloudBreak and Ambari Blue Prints)

Deploying Hortonworks HDP on Microsoft Azure Video

Step by step guide - TechNet blog

Few specific questions :

- Should we place each of the node in a separate cloud service or all nodes in a single cloud service?

- Should we setup HDP on all the nodes (18 nodes) and setup a Windows Server with Active directory for name resolution as show in the TechNet blog? Can we not have HDP cluster running without jumpbox?

- Do we need to build a separate VPC?

- Is there any documentation in place which talks about step by step setting up?

- What are the best practices around?

[#2]

@Neeraj Sabharwal There is a new doc, re: HDP deployment on Azure: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_HDP_AzureSetup/content/ch_HDP_AzureSetup....

[#3]

@Smart Solutions

If you use "Resource Manager" for your deployment model, each vm should get a separated cloud service. I recommend that you use a sepateted cloud service for each node.

Here are some informations about the deployment methods: https://azure.microsoft.com/en-gb/documentation/articles/resource-manager-deployment-model/

We also work with reserved ip's on ambari node and on the master nodes. This have benefits if your comapny has strong security concepts like port forwarding.