what are steps of easiest way to create an HDP cluster on raw HP servers

[#1]

Hi,

I am new in making private clouds and hadoop ecosystem. We have three HP servers without ESX or openstack, and want to create a private cloud and then have an HDP cluster with Ambari. It would be appreciated, If somebody could explain the steps of creating a cloud and hadoop cluster by using (ESXi, openstack, cloudbreak, Ambari and ....). there are some tutorials to do this, but they are confusing to me. some of our requirements are as follows:

1- we must have at least 100TB HDFS capacity.

2- we should be able to add new nodes for increasing the capacity of HDFS and processing power.

3- we should be able to share the HDP cluster with multiple users. each user should be able to run own map/reduce job.

Sincerely,

[#2]

@Ali Mohammadi Shanghoshabad

You have this option which covers all the 3 requirements:

1. Install and setup an OpenStack version compatible with Cloudbreak
2. Install Cloudbreak Deployer and start Cloudbreak
3. Create an OpenStack credential (which connects your OpenStack account with Cloudbreak) and infrastructure templates in Cloudbreak
4. Create or reuse a blueprints matching your HDP workload type
5. Launch cluster(s)
6. Configure autoscale and add users via Ambari as needed

Hope this helps!

[#3]

@pdarvasi

Thanks for your valuable and useful suggestion. Actually, my major problem is step one.

I need to have openstack on current hp servers and also need to extend it whenever it is necessary. shall I install ESXi and then create a VM and after that install openstack? or I can install openstack on raw servers and then can create a extendable private cloud without ESXi. In fact, I am confusing with creating clusters. For example, I thought, I have to manage a cluster of servers on Vsphere, a cluster of nodes on openstack, and then a hadoop cluster on Ambari. It's still confusing for me. It would be appreciated if you could give me more information or introduce some tutorials.

thanks a million,

ALi

[#4]

@Ali Mohammadi Shanghoshabad

That first step has very little to do with Hortonworks, that's why I was not elaborating on it in detail.

As a rule of thumb, if you can go with KVM hypervisor, you should go with it.

The reasons are:

1. It os the only officially supported hypervisor for OpenStack, most OpenStack development is done with the KVM hypervisor
2. Cloudbreak supports KVM for OpenStack.
3. The reference architecture contains KVM as well.

Hope this helps!

[#5]

@pdarvas

thanks, I am going to check it.