HDP on private cloud (AWS, Azure)

[#1]

What are my options to deploy HDP on the private cloud of both AWS and Azure?

As per my understanding,

AWS

Private Cloud in the AWS world mean VPC. For IaaS on VPC, Can i deploy using CloudBreak?

Do i have the option of HDCloud in AWS VPC?

Azure

Private cloud in the Azure mean hosted private cloud on-premise? Or there is also a flavor similar to VPC?

Do i get HDInsight, HDP in Azure marketplace in private cloud?

Can i deploy using CloudBreak?

In both the AWS and Azure cases, what are my storage options?

How do i move data from on-premise to private cloud?

[#2]

@learninghuman 1. From IaaS perspective you can deploy HDP in Private or Public cloud using Cloudbreak. It supports both Azure & AWS.

2. Yes you HDCloud for AWS have option to setup new VPC or cn work with existing one.

3. From Azure perspective consider HDInsight as your PaaS. Also you have option to setup HDP in priavate or public cloud as mentioned in point 1. Azure marketplace have HDP standard and Hortonworks sandbox options.

4. You can using HDF powered by NiFi for the data movement in hybrid architecture. NiFi supports site to site protocol which can be leverage to run the nifi instances on prem and cloud to exchange the data.

[#3]

@milind pandit When you say private cloud, i assume that its VN in the Azure world and VPC in the AWS world. Is that right?

[#4]

Thats correct, you can setup the cluster in a dedicated VNet in Azure and VPC in aws.

[#5]

@milind pandit so to summarize (with a question)

AWS

HDCloud can be set only on the private cloud(not in the public cloud)

HDP cluster on IaaS can be created on both the public and the private cloud

Azure

HDInsights - Its a manage service. Can it be created on both the private and the public cloud?

HDP in the marketplace - can be created on both the public and the private cloud

HDP cluster on IaaS - can be created on both the public and the private cloud

[#6]

@learninghuman

HDInsight does not require a secure Virtual Network. However, since it is a Managed Service, if you need to install HDInsight into a secured Virtual Network, you must allow inbound access over port 443 for the following IP addresses, which allow Azure to manage the HDInsight cluster.

• 168.61.49.99
• 23.99.5.239
• 168.61.48.131
• 138.91.141.162

[#7]

You can use Cloudbreak for using existing VPC-s or Virtual Networks in both AWS and Azure.

AWS

Both HDC and Cloudbreak support the usage of an existing VPC.

HDC support is documented here

And you have the following options in Cloudbreak:

• Create a new VPC and a new subnet: Every time a cluster is created with this kind of network setup a new VPC and a new subnet with the specified IP range will be created for the instances on AWS.
• Create a new subnet in an existing VPC: Use this kind of network setup if you already have a VPC on AWS where you'd like to put the Cloudbreak created cluster but you'd like to have a separate subnet for it. This setup is only supported for basic VPCs, where an Internet Gateway is configured and instances can have public IP addresses to access the Internet. If you have a specific VPC setup (VGW, NAT, private subnets, etc..) then only the third option can be used.
• Use an existing subnet in an existing VPC: Use this kind of network setup if you have an existing VPC with one or more subnets on AWS and you'd like to start the instances of a cluster in one - or more - of those subnets. Use this setup if you have a specific VPC setup: you should first create the subnet(s) directly through AWS and provide their IDs here. The subnets could be even in different availability zones and you can set a single or a comma separated list of subnets in the 'Subnet Identifier' field.

You can find the rest of the documentation here.

Azure

Virtual network is the analogous term to AWS VPC in Azure and one can set up their private Virtual Networks in Azure.

You have the following options in Cloudbreak:

• Create a new virtual network and a new subnet: Every time a cluster is created with this kind of network setup a new virtual network and a new subnet with the specified IP range will be created for the instances on Azure.
• Use an existing subnet in an existing virtual network: Use this kind of network setup if you have an existing virtual network with one or more subnets on Azure and you'd like to start the instances of a cluster in one of those subnets.

You can find the rest of the documentation here. The latest RC versions of Cloudbreak support the usage of only private IP-s in a private virtual network.

You can upload data using various methods in Azure (mentioned here), most of them applies for both HDP and HDInsight.

There is support for WASB storage (with or without DASH) and ADLS storage support is in the near term roadmap as well.

Let me know if you have additional questions!

[#8]

@pdarvasi, Thanks. Few more questions

1. Is is possible to have private cloud hosted on-premise using Azure (not the virtual network) and use the services of HDInsights, HDP from market place?

2. From the link you gave, it talks about blob storage. Is that same as WASB?

3. What about HDInsight and HDP from MarketPlace on the virtual network? I hope i can still do that.

4. How do i get the HDP running in the hybrid architecture in both Azure and AWS? By hybrid architecture, i mean running jobs on 2 situations i. on-premise, cloud (public as well private (VPC, VN) ) combination and ii. public, private (VPC, VN) combination

[#9]

If you would like to have quick responses from Cloudbreak team, you might use "cloudbreak" tag as well. :)