Deploy HDP Sandbox on AWS

[#1]

Note: HDP 2.5 is used, but can be updated for HDP 2.6 and beyond.

1. Boot an Amazon Linux instance with at least 16GB of RAM
2. Execute the following. It will take a while

## install docker
sudo yum update -y
sudo yum install -y docker

## fix docker for importing the large sandbox image
sed -i.backup 's/\(^OPTIONS=.*\)"$/\1 --storage-opt=dm.basesize=20G"/' /etc/sysconfig/docker

## start docker
sudo service docker start

## confirm docker is working
sudo usermod -a -G docker ec2-user
docker info

## download docker image
curl -O http://hortonassets.s3.amazonaws.com/2.5/HDP_2.5_docker.tar.gz

## load docker image
docker load -i HDP_2.5_docker.tar.gz
## confirm image is available
docker images

## get sandbox docker startup script
curl -O https://raw.githubusercontent.com/hortonworks/tutorials/hdp-2.5/tutorials/hortonworks/hortonworks-sa...

## start sandbox
bash start_sandbox.sh

## configure to start at boot
echo "bash /root/start_sandbox.sh" >> /etc/rc.local

## Print the URL for accessing the Sandbox
echo -e "##\nAccess the Sandbox at:\nhttp://$(curl -sS4 icanhazip.com):8888\n##"

[#2]

The very first command fails:

sudo yum update -y

Most references suggest that outbound interfaces are not properly set-up on the instance. How one should set them up in order to get this script to work?

[#3]

Updated link for HDP 2.6.1 docker image and HDP startup script for Linux:

https://downloads-hortonworks.akamaized.net/sandbox-hdp-2.6.1/HDP_2_6_1_docker_image_28_07_2017_14_4...

https://raw.githubusercontent.com/hortonworks/data-tutorials/master/tutorials/hdp/sandbox-deployment...

It looks like the yum repository with docker is not enabled by default on AWS RHEL EC2 instances. If the yum update works, but you are unable to install docker, try the following to enable the right extras repository:

sudo yum install yum-utils
sudo yum-config-manager --enable rhui-REGION-rhel-server-extras
sudo yum install docker

[#4]

It also looks like the command to expand the base image from 10GB to 20GB may not work on later versions of docker because the config file is ephemeral. This worked for me, instead (if you run into the dreaded "no space left on device" error.

docker daemon --storage-opt dm.basesize=20G

I used 30G, just to be safe, but the image size for HDP 2.6.1 looks to be about 12.13 GB total, so 20 should do.

[#5]

One last note: You'll need to update the security rules for your AWS instance. Initially, you have to add a custom TCP rule to allow traffic on port 8888 (usually - Sean's last command will tell you the port). Once you have access, the sandbox portal will give you the ports for all the other services that need to be opened up.

[#6]

I'm getting consistent "overlay2: Unknown option dm.basesize" in the logfile when I follow your instructions. I've also tried your storage-opt hack below, which produces the same error. I'm using Amazon Linux, and have plenty of storage config'd. Any ideas ?

[#7]

(Responding to myself) I finally managed to get everything running, despite a very brittle and overly complicated process. See my gist: https://gist.github.com/renodino/5bd8adc661bd2f00a5b662ed0713fd0f

[#8]

The startup script link is no longer available, and I have searched quite a bit to find it without success. Does anyone have an idea where it has moved to?

[#9]

The tutorial from which the script was linked has been deprecated on github, but there is a similar set of instructions here:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.5.1/content/aws-launch/index.html