Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar

Note: HDP 2.5 is used, but can be updated for HDP 2.6 and beyond.

  1. Boot an Amazon Linux instance with at least 16GB of RAM
  2. Execute the following. It will take a while
## install docker
sudo yum update -y
sudo yum install -y docker

## fix docker for importing the large sandbox image
sed -i.backup 's/\(^OPTIONS=.*\)"$/\1 --storage-opt=dm.basesize=20G"/' /etc/sysconfig/docker

## start docker
sudo service docker start

## confirm docker is working
sudo usermod -a -G docker ec2-user
docker info

## download docker image
curl -O http://hortonassets.s3.amazonaws.com/2.5/HDP_2.5_docker.tar.gz

## load docker image
docker load -i HDP_2.5_docker.tar.gz
## confirm image is available
docker images

## get sandbox docker startup script
curl -O https://raw.githubusercontent.com/hortonworks/tutorials/hdp-2.5/tutorials/hortonworks/hortonworks-sa...

## start sandbox
bash start_sandbox.sh

## configure to start at boot
echo "bash /root/start_sandbox.sh" >> /etc/rc.local

## Print the URL for accessing the Sandbox
echo -e "##\nAccess the Sandbox at:\nhttp://$(curl -sS4 icanhazip.com):8888\n##"

Deploying!

13,262 Views
Comments
avatar
Contributor

The very first command fails:

sudo yum update -y

Most references suggest that outbound interfaces are not properly set-up on the instance. How one should set them up in order to get this script to work?

avatar
Expert Contributor

Updated link for HDP 2.6.1 docker image and HDP startup script for Linux:

https://downloads-hortonworks.akamaized.net/sandbox-hdp-2.6.1/HDP_2_6_1_docker_image_28_07_2017_14_4...
https://raw.githubusercontent.com/hortonworks/data-tutorials/master/tutorials/hdp/sandbox-deployment...

It looks like the yum repository with docker is not enabled by default on AWS RHEL EC2 instances. If the yum update works, but you are unable to install docker, try the following to enable the right extras repository:

sudo yum install yum-utils
sudo yum-config-manager --enable rhui-REGION-rhel-server-extras
sudo yum install docker
avatar
Expert Contributor

It also looks like the command to expand the base image from 10GB to 20GB may not work on later versions of docker because the config file is ephemeral. This worked for me, instead (if you run into the dreaded "no space left on device" error.

docker daemon --storage-opt dm.basesize=20G

I used 30G, just to be safe, but the image size for HDP 2.6.1 looks to be about 12.13 GB total, so 20 should do.

avatar
Expert Contributor

One last note: You'll need to update the security rules for your AWS instance. Initially, you have to add a custom TCP rule to allow traffic on port 8888 (usually - Sean's last command will tell you the port). Once you have access, the sandbox portal will give you the ports for all the other services that need to be opened up.

avatar
New Contributor

I'm getting consistent "overlay2: Unknown option dm.basesize" in the logfile when I follow your instructions. I've also tried your storage-opt hack below, which produces the same error. I'm using Amazon Linux, and have plenty of storage config'd. Any ideas ?

avatar
New Contributor

(Responding to myself) I finally managed to get everything running, despite a very brittle and overly complicated process. See my gist: https://gist.github.com/renodino/5bd8adc661bd2f00a5b662ed0713fd0f

avatar
New Contributor

The startup script link is no longer available, and I have searched quite a bit to find it without success. Does anyone have an idea where it has moved to?

avatar
Expert Contributor

The tutorial from which the script was linked has been deprecated on github, but there is a similar set of instructions here:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.5.1/content/aws-launch/index.html

Version history
Last update:
‎05-19-2017 04:12 PM
Updated by:
Contributors