What are my options to deploy HDP on the private cloud of both AWS and Azure?
As per my understanding,
Private Cloud in the AWS world mean VPC. For IaaS on VPC, Can i deploy using CloudBreak?
Do i have the option of HDCloud in AWS VPC?
Private cloud in the Azure mean hosted private cloud on-premise? Or there is also a flavor similar to VPC?
Do i get HDInsight, HDP in Azure marketplace in private cloud?
Can i deploy using CloudBreak?
In both the AWS and Azure cases, what are my storage options?
How do i move data from on-premise to private cloud?
@learninghuman 1. From IaaS perspective you can deploy HDP in Private or Public cloud using Cloudbreak. It supports both Azure & AWS.
2. Yes you HDCloud for AWS have option to setup new VPC or cn work with existing one.
3. From Azure perspective consider HDInsight as your PaaS. Also you have option to setup HDP in priavate or public cloud as mentioned in point 1. Azure marketplace have HDP standard and Hortonworks sandbox options.
4. You can using HDF powered by NiFi for the data movement in hybrid architecture. NiFi supports site to site protocol which can be leverage to run the nifi instances on prem and cloud to exchange the data.
@milind pandit so to summarize (with a question)
HDCloud can be set only on the private cloud(not in the public cloud)
HDP cluster on IaaS can be created on both the public and the private cloud
HDInsights - Its a manage service. Can it be created on both the private and the public cloud?
HDP in the marketplace - can be created on both the public and the private cloud
HDP cluster on IaaS - can be created on both the public and the private cloud
HDInsight does not require a secure Virtual Network. However, since it is a Managed Service, if you need to install HDInsight into a secured Virtual Network, you must allow inbound access over port 443 for the following IP addresses, which allow Azure to manage the HDInsight cluster.
You can use Cloudbreak for using existing VPC-s or Virtual Networks in both AWS and Azure.
Both HDC and Cloudbreak support the usage of an existing VPC.
HDC support is documented here
And you have the following options in Cloudbreak:
You can find the rest of the documentation here.
Virtual network is the analogous term to AWS VPC in Azure and one can set up their private Virtual Networks in Azure.
You have the following options in Cloudbreak:
You can find the rest of the documentation here. The latest RC versions of Cloudbreak support the usage of only private IP-s in a private virtual network.
You can upload data using various methods in Azure (mentioned here), most of them applies for both HDP and HDInsight.
There is support for WASB storage (with or without DASH) and ADLS storage support is in the near term roadmap as well.
Let me know if you have additional questions!
@pdarvasi, Thanks. Few more questions
1. Is is possible to have private cloud hosted on-premise using Azure (not the virtual network) and use the services of HDInsights, HDP from market place?
2. From the link you gave, it talks about blob storage. Is that same as WASB?
3. What about HDInsight and HDP from MarketPlace on the virtual network? I hope i can still do that.
4. How do i get the HDP running in the hybrid architecture in both Azure and AWS? By hybrid architecture, i mean running jobs on 2 situations i. on-premise, cloud (public as well private (VPC, VN) ) combination and ii. public, private (VPC, VN) combination