How to convert the HDP Sandbox into a Vagrant Box on Mac OS X

[#1]

Objective

This tutorial is intended to describe the process of converting the Hortonworks HDP 2.5 Technical Preview sandbox, which is a VirtualBox virtual machine, into a Vagrant box.

Overview

Given the limited resources of a single virtual machine, it can be difficult to run and manage multiple demo environments within a single virtual machine. It is a common desire to add additional software components to the sandbox. Adding additional components often means having to manually manage port forwarding rules in VirtualBox or you have to add an additional network interface to the virtual machine to directly access it via an IP address using either host-only or bridged networking. Given the steps involved, it is common to do this once and try to reuse the same virtual machine. This may not always be ideal.

Using Vagrant, you can quickly create a clean sandbox with a simple command like vagrant init HDP25TP && vagrant up. Vagrant makes it easy to connect to the virtual machine via vagrant ssh. Vagrant will also auto-mount the host vagrant/project directory to /vagrant on the virtual machine. This makes it incredibly easy to add files or data to the virtual machine by simply copying the data to the mounted directory.

The addition of the Vagrant plugin vagrant-vbguest automates the installation and maintenance of the VirtualBox Guest Additions software. Out of the box, the sandbox does not provide the VirtualBox Guest Additions. You can add the additions yourself, however, it is easy for the additions to stop working whenvever you update the virtual machine operating system kernel via yum upgrade or whenever you update the installed version of VirtualBox. The plugin will automatically handle this for you everytime you run vagrant up.

Scope

This tutorial was created and validated on Mac OS X. However, the overall process should work for Windows. This tutorial has been tested with the following configuration:

• Mac OS X 10.11.5
• VirtualBox 5.0.24
• Vagrant 1.8.4
• vagrant-vaguest 0.12.0

and

• Mac OS X 10.11.5
• VirtualBox 5.1.4
• Vagrant 1.8.5
• vagrant-vbguest 0.13.0

Note: If you upgrade VirtualBox from version 5.0.x to 5.1.x, you must also upgrade Vagrant from 1.8.4 to 1.8.5.

Prerequisites

• VirtualBox for Mac OS X installed
  • VirtualBox Link
• Vagrant software for Mac OS X installed
  • Vagrant Link
• Vagrant plugin vagrant-vbguest installed
  • Vagrant Plugin Link
• Download Hortonworks HDP 2.5 Technical Preview Sandbox
  • Sandbox Link

Steps

Prepare The Sandbox Virtual Machine

Before creating the Vagrant box from the HDP sandbox, there are some configuration changes we need to complete.

• Import the Hortonworks HDP 2.5 Technical Preview sandbox OVA file into VirtualBox. You should be able to use the GUI or the command line.
  • NOTE: When I use the command line, the eth0 network device would not work properly. I believe this is a VirtualBox issue as this problem does not happen when you use the GUI to import the OVA file.

  $ VBoxManage import ./"Hortonworks Sandbox with HDP 2.5 Technical Preview.ova" --vsys 0 --eula accept
  0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
  Interpreting /Users/username/Downloads/VirtualBox Appliances/./Hortonworks Sandbox with HDP 2.5 Technical Preview.ova...
  OK.
  Disks:
  vmdisk1   52428800000 -1  http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized   Hortonworks Sandbox with HDP 2.5 Technical Preview-disk1.vmdk   -1  -1
  Virtual system 0:
  0: Suggested OS type: "RedHat_64"
      (change with "--vsys 0 --ostype <type>"; use "list ostypes" to list all possible values)
  1: Suggested VM name "Hortonworks Sandbox with HDP 2.5 Technical Preview"
      (change with "--vsys 0 --vmname <name>")
  2: Number of CPUs: 4
      (change with "--vsys 0 --cpus <n>")
  3: Guest memory: 8192 MB
      (change with "--vsys 0 --memory <MB>")
  4: USB controller
      (disable with "--vsys 0 --unit 4 --ignore")
  5: Network adapter: orig NAT, config 2, extra slot=0;type=NAT
  6: IDE controller, type PIIX4
      (disable with "--vsys 0 --unit 6 --ignore")
  7: IDE controller, type PIIX4
      (disable with "--vsys 0 --unit 7 --ignore")
  8: Hard disk image: source image=Hortonworks Sandbox with HDP 2.5 Technical Preview-disk1.vmdk, target path=/Users/username/VirtualBox VMs/Hortonworks Sandbox with HDP 2.5 Technical Preview/Hortonworks Sandbox with HDP 2.5 Technical Preview-disk1.vmdk, controller=6;channel=0
      (change target path with "--vsys 0 --unit 8 --disk path";
      disable with "--vsys 0 --unit 8 --ignore")
  0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
  Successfully imported the appliance.

• Start the imported virtual machine. You can use the GUI or the command line.

  $ VBoxManage startvm "Hortonworks Sandbox with HDP 2.5 Technical Preview"
  Waiting for VM "Hortonworks Sandbox with HDP 2.5 Technical Preview" to power on...
  VM "Hortonworks Sandbox with HDP 2.5 Technical Preview" has been successfully started.

• The sandbox requires you to reset the root password upon initial login. Change the password when prompted to a secure password. CentOS will not let you use “hadoop” at this stage.
• For convenience, you can reset the password back to “hadoop” so that it is consistent with the VM banner displayed after bootup. This is very handy after a couple of months has passed and you can’t remember what you set the password to. Change the password back to “hadoop”

  # passwd

• For a new sandbox, the Ambari admin user password needs to be reset to enable the admin account. I recommend you set it something easy to remember like ‘admin’ or ‘password’.
  • NOTE: You would not use a simple password for production systems.

  # ambari-admin-password-reset

• Add the vagrant user. This user is what Vagrant expects by default.

  # useradd vagrant -d /home/vagrant

• Set the vagrant user password to “vagrant”.

  # passwd vagrant

• The vagrant user needs to be able to run sudo.
  • NOTE: You can also create a file named “vagrant” in /etc/sudoers.d, however, the requiretty line is in /etc/sudoers so it may be more convenient to simply modify /etc/sudoers.

  # visudo

  • Add an additional “env_keep” setting after the existing “env_keep” settings.

  Defaults env_keep += "SSH_AUTH_SOCK"

  • Add vagrant no password setting at the end of the file.

  vagrant ALL=(ALL) NOPASSWD:ALL

  • Ensure the “Defaults requiretty” line is commented out.

  #Defaults requiretty

  • Save the file

  !wq

• Now switch to the vagrant user

  # su - vagrant

• Test sudo as the vagrant user. You should not be prompted for a password.

  $ sudo ls

• Create a .ssh directory for the vagrant user

  $ cd
  $ mkdir .ssh

• Download the vagrant public key and save it as “authorized_keys” in the .ssh directory

  $ wget --no-check-certificate https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub -O /home/vagrant/.ssh/authorized_keys

• Change the permissions of the .ssh directory and authorized_key files or ssh will not work properly.

  $ chmod 0700 .ssh
  $ chmod 0600 .ssh/authorized_keys

• Shutdown the virtual machine

  $ sudo init 0

Create the Vagrant Box

If everything worked correctly, you are ready to create a Vagrant box using the modified virtual machine.

• Make a directory for creating the box

  $ mkdir create-sandbox && cd create-sandbox

• Initialize a default Vagrantfile

  $ vagrant init

• Edit the Vagrantfile and replace everythng in the file with the following:

  # -*- mode: ruby -*-
  # vi: set ft=ruby :
  Vagrant.configure("2") do |config|
    config.vm.box = "HDP25TP" # name of the base vagrant box, should line up with the --name option used with vagrant box add command
    config.vm.box_check_update = false # don't update the box version on boot
    config.vm.network :private_network, type: "dhcp", :adapater=>2 # add a private network to avoid having to forward ports
    config.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--cpuexecutioncap", "75"] # set maximum host cpu usage to 75%
      vb.cpus = "4" # set the number of cpus to 4
      vb.memory = "8192" # set the memory to 8GB
    end
  end

• Get the unique ID of the virtual machine:

  $ VBoxManage list vms
  "Hortonworks Sandbox HDP 2.4 IoT Demo" {c9db5b78-554f-41d5-beba-a27d54ee42db}
  "Hortonworks Sandbox HDP 2.4 NiFi Demo" {3abe96ef-258b-4395-b9bd-590b1685132d}
  "CentOS 6.8 Minimal" {bc5178e9-061f-47c8-897d-c424aa3a50de}
  "hdp24_ldap" {b8e1f3ad-6c06-4e8a-849a-0249611a607a}
  "hdp24_sandbox" {125e8108-21e6-48ce-87f8-a065a50cde3a}
  "default" {3a471444-fb24-40d3-9954-c53d9ea03895}
  "Hortonworks Sandbox with HDP 2.5 Technical Preview" {e250b938-bece-414a-ba8a-77972959b5f8}

• Look for the ID value inside the {} for the virtual machine named “Hortonworks Sandbox with HDP 2.5 Technical Preview”

  "Hortonworks Sandbox with HDP 2.5 Technical Preview" {e250b938-bece-414a-ba8a-77972959b5f8}

• Package the vagrant box using the ID from the previous step.
  • NOTE: Specifying the --vagrantfile option ensures the settings are applied at boot without having to specify them in every Vagrantfile you create. The --output option specifies the name of the box file.

  $ vagrant package --base e250b938-bece-414a-ba8a-77972959b5f8 --output hdp25tp.box --vagrantfile Vagrantfile
  ==> e250b938-bece-414a-ba8a-77972959b5f8: Clearing any previously set forwarded ports...
  ==> e250b938-bece-414a-ba8a-77972959b5f8: Exporting VM...
  ==> e250b938-bece-414a-ba8a-77972959b5f8: Compressing package to: /Users/username/Vagrant/create-sandbox/hdp25tp.box
  ==> e250b938-bece-414a-ba8a-77972959b5f8: Packaging additional file: Vagrantfile

• Add the newly created box to Vagrant
  • NOTE: The --name option can be anything you like. Use something intuitive.

  $ vagrant box add hdp25tp.box --name HDP25TP
  ==> box: Box file was not detected as metadata. Adding it directly...
  ==> box: Adding box 'HDP25TP' (v0) for provider:
  box: Unpacking necessary files from: file:///Users/username/Vagrant/create-sandbox/hdp25tp.box
  ==> box: Successfully added box 'HDP25TP' (v0) for 'virtualbox'!

• Create a demo project directory

  $ cd /vagrant-projects
  $ mkdir demo1 && cd demo1

• Initialize the Vagrant environment
  • NOTE: If you specify the name used with the --name option above, you won’t have to edit the Vagrantfile.

  $ vagrant init HDP25TP
  A `Vagrantfile` has been placed in this directory. You are now
  ready to `vagrant up` your first virtual environment! Please read
  the comments in the Vagrantfile as well as documentation on
  `vagrantup.com` for more information on using Vagrant.

• Start the vagrant machine
  • NOTE: During the first boot of a new vagrant virtual machine, you will notice the VirtualBox Guest Additions will be installed.

  $ vagrant up
  Bringing machine 'default' up with 'virtualbox' provider...
  ==> default: Importing base box 'HDP25TP'...
  ==> default: Matching MAC address for NAT networking...
  ==> default: Setting the name of the VM: demo1_default_1468377822096_63592
  ==> default: Clearing any previously set network interfaces...
  ==> default: Preparing network interfaces based on configuration...
  default: Adapter 1: nat
  default: Adapter 2: hostonly
  ==> default: Forwarding ports...
  default: 22 (guest) => 2222 (host) (adapter 1)
  ==> default: Running 'pre-boot' VM customizations...
  ==> default: Booting VM...
  ==> default: Waiting for machine to boot. This may take a few minutes...
  default: SSH address: 127.0.0.1:2222
  default: SSH username: vagrant
  default: SSH auth method: private key
  default:
  default: Vagrant insecure key detected. Vagrant will automatically replace
  default: this with a newly generated keypair for better security.
  default:
  default: Inserting generated public key within guest...
  default: Removing insecure key from the guest if it's present..
  ==> default: Machine booted and ready!
  [default] No installation found.
  Loaded plugins: fastestmirror, riorities
  Setting up Install Process
  Determining fastest mirrors
  * base: mirror.vcu.edu
  * epel: lug.mtu.edu
  * extras: mirror.vcu.edu
  * updates: mirrors.cat.pdx.edu
  Package kernel-devel-2.6.32-642.1.1.el6.x86_64 already installed and latest version
  Package gcc-4.4.7-17.el6.x86_64 already installed and latest version
  Package binutils-2.20.51.0.2-5.44.el6.x86_64 already installed and latest version
  Package 1:make-3.81-23.el6.x86_64 already installed and latest version
  Package 4:perl-5.10.1-141.el6_7.1.x86_64 already installed and latest version
  Package bzip2-1.0.5-7.el6_0.x86_64 already installed and latest version
  Nothing to do
  Copy iso file /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
  Installing Virtualbox Guest Additions 5.0.24 - guest version is unknown
  Verifying archive integrity... All good.
  Uncompressing VirtualBox 5.0.24 Guest Additions for Linux............
  VirtualBox Guest Additions installer
  Copying additional installer modules ...
  Installing additional modules ...
  Removing existing VirtualBox non-DKMS kernel modules[ OK ]
  Building the VirtualBox Guest Additions kernel modules
  Building the main Guest Additions module[ OK ]
  Building the shared folder support module[ OK ]
  Building the graphics driver module[ OK ]
  Doing non-kernel setup of the Guest Additions[ OK ]
  Starting the VirtualBox Guest Additions Installing the Window System drivers
  Could not find the X.Org or XFree86 Window System, skipping.
  [ OK ]
  ==> default: Checking for guest additions in VM...
  ==> default: Configuring and enabling network interfaces...
  ==> default: Mounting shared folders...
  default: /vagrant => /Users/username/Vagrant/demo1

• SSH into the vagrant machine

  $ vagrant ssh

  • Note: There is a known issue with Vagrant 1.8.5 : https://github.com/mitchellh/vagrant/issues/7610. The first time you boot a VM using vagrant up, vagrant will not be able to connect when you "vagrant ssh". This is because vagrant replaces the temporary ssh key with a new one and the permissions are 664 instead of 600. To fix this, log into the VM using the VirtualBox interface and change the permission of the file in /home/vagrant/.ssh/authorized_keys to 600. This issue will be fixed in Vagrant 1.8.6.

• Verify the network interfaces:
  • NOTE: You should note there are two ethernet devices. The first device, eth0, is configured with NAT for internet access. The second device, eth1, is configured with host-only networking. Note the IP address of of eth1. This is the IP address you will use to interact with the sandbox web interfaces instead of setting up all of the port forwarding. The IP address for eth1 will depend on your specific VirtualBox settings. You can modify these setting to change the host-only network to use different IP addresses.

  $ ifconfig -a
  eth0      Link encap:Ethernet  HWaddr 08:00:27:8B:B4:D2
            inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:725 errors:0 dropped:0 overruns:0 frame:0
            TX packets:494 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:80100 (78.2 KiB)  TX bytes:63763 (62.2 KiB)
            Interrupt:19 Base address:0xd020
  eth1      Link encap:Ethernet  HWaddr 08:00:27:C8:EB:D8
            inet addr:172.28.128.3  Bcast:172.28.128.255  Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:3 errors:0 dropped:0 overruns:0 frame:0
            TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:1770 (1.7 KiB)  TX bytes:1266 (1.2 KiB)
  lo        Link encap:Local Loopback
            inet addr:127.0.0.1  Mask:255.0.0.0
            UP LOOPBACK RUNNING  MTU:65536  Metric:1
            RX packets:3 errors:0 dropped:0 overruns:0 frame:0
            TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:165 (165.0 b)  TX bytes:165 (165.0 b)

• On the host machine, ls the vagrant project directory

  $ ls /vagrant-projects/demo1
  Vagrantfile

• Create a test file on the host machine

  touch /vagrant-projects/demo1/local-file.txt

• On the vagrant virtual machine, verify you can see the new file

  $ ls /vagrant
  Vagrantfile local-file.txt

• Create a test file on the Vagrant virtual machine

  $ touch /vagrant/vm-file.txt

• On the host machine, verify you can see the new file

  $ ls /vagrant-projects/demo1/
  Vagrantfile local-file.txt vm-file.txt

• Exit the vagrant virtual machine

  $ exit

• Shutdown the Vagrant virtual machine

  $ vagrant halt

During the Vagrant setup process, Vagrant will have cleared the forwarded ports for the Vagrant virtual machine. This also cleared the ports of the sandbox virtual machine we initially imported into VirtualBox. The sandbox virtual machine is no longer needed, so you can delete it from VirtualBox.

Review

If all of the steps completed successfully, you now have the ability to create demo virtual machines as needed with a couple of simple commands.

$ mkdir <demo directory> && cd <demo directory>
$ vagrant init HDP25TP
$ vagrant up
$ vagrant ssh

[#2]

If you upgrade VirtualBox to 5.1.x, then you also need to upgrade Vagrant to 1.8.5. Otherwise, Vagrant will complain about an unsupported version of VirtualBox.

[#3]

Awesome!

The new HDP 2.5 sandbox comes without wget preinstalled but

$ curl -kL https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub -o /home/vagrant/.ssh/authorized_keys

works just as well.