Setup Hortonworks Data Platform using Vagrant, VirtualBox and Ambari

[#1]

Objective

Deploy a 4-node HDP 2.4.2 cluster with Apache Ambari 2.2.2, Vagrant and VirtualBox on OS X host. This is helpful for development and proof of concepts.

Scope

This approach has been tested on OS X host, but it should work on all supported Vagrant and VirtualBox environments.

Pre-requisites

• Minimum 9 GB of RAM for the HDP 2.4.2 cluster
• Download and install Vagrant for your host OS: https://www.vagrantup.com/downloads.html
• Download and install VirtualBox for your host OS: https://www.virtualbox.org/wiki/Downloads
• Download and install git client for your host
• Open a command shell and change to the folder where you plan to clone the github repository
• Clone the following git repository

git clone https://github.com/cstanca1/hdp2_4_2-vagrant.git

Create and Start VMs

Change directory to /hdp_2.4.2-vagrant, the folder that includes Vagrantfile and create a /data folder:

mkdir data

This /data folder will be needed for guest VMs to share with the host.

Vagrant (via Vagrantfile) is configured to use Centos 6.7 as the base box and includes the pre-requisites for installing HDP. 4 VMs will be created: 1 Ambari Server (ambari1), 1 Hadoop master (master1) and 2 slaves (slave1, slave2).

vagrant up ambari1

Install and Setup Ambari Server

Set a Local Reference to a Remote Ambari Repo

vagrant ssh ambari1

sudo su -

cd /etc/yum.repos.d

wget http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari.repo

Setup SSH Access and Starting the Other 3 VMs

Add at the same path with files downloaded from the repoosity, your id_rsa and id_rsa.pub keys (seehttps://wiki.centos.org/HowTos/Network/SecuringSSH section 7 for instructions on CentOS). You could perform these steps on ambari1 VM and copy these two files to your /vagrant_data folder which shares data between guest and host. Only after you copy those two files, start the other three VMs:

vagrant up master1

vagrant up slave1

vagrant up slave2

Install Ambari Server

yum install ambari-server

Setup Ambari Server

Run the setup command to configure your Ambari Server, Database, JDK, LDAP, and other options:

ambari-server setup

Start Ambari Server

ambari-server start

Deploy Cluster using Ambari Web UI

Open up a web browser and go to:

http://ambari1:8080

Log in with username admin and password admin and follow on-screen instructions, using hosts created and selecting services of interest. For more details, see "Automated Install" at: https://docs.hortonworks.com/HDPDocuments/Ambari/Ambari-2.2.2.0/index.html

[#2]

For full guide on the ambari quickstart on vagrant follow the apache doc: https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide

[#3]

This is great. It will save me lots of time. I am trying this on an Ubuntu host, not a Mac, and everything is fine until I get down to trying to access ambari1 through the web. I don't have ambari1 in DNS anywhere.

I can "vagrant ssh ambari1" and find the IP address, but presumably that wont let me install without FQDNs.

Any ideas? Thanks again.

[#4]

To partly answer my own question... Hadoop doesn't need DNS if all the machines are already in the hosts file. So at some point in the above the /etc/hosts file was populated with

192.168.0.11 ambari1.mycluster ambari1

192.168.0.12 master1.mycluster master1

192.168.0.21 slave1.mycluster slave1

192.168.0.22 slave2.mycluster slave2

[#5]

You use the phrase "Create a Local Ambari Repository". that is not what you are doing. You are merely pointing you machines to look at an external repository. It would be great if you did create a local ambari repository - but that would require a lot more explanation. I had a quick look at Nexus for this, but found it was not trivial to set up for my purposes.

[#6]

@Alex McLintock

You are correct. This is a local reference to a remote repo. Here is a reference for an actual local repo with no Internet access, but it gets complicated with Vagrant: https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_setting_up_a...