Note: HDP 2.5 is used, but can be updated for HDP 2.6 and beyond. Boot an Amazon Linux instance with at least 16GB of RAM Execute the following. It will take a while ## install docker sudo yum update -y sudo yum install -y docker ## fix docker for importing the large sandbox image sed -i.backup 's/\(^OPTIONS=.*\)"$/\1 --storage-opt=dm.basesize=20G"/' /etc/sysconfig/docker ## start docker sudo service docker start ## confirm docker is working sudo usermod -a -G docker ec2-user docker info ## download docker image curl -O http://hortonassets.s3.amazonaws.com/2.5/HDP_2.5_docker.tar.gz ## load docker image docker load -i HDP_2.5_docker.tar.gz ## confirm image is available docker images ## get sandbox docker startup script curl -O https://raw.githubusercontent.com/hortonworks/tutorials/hdp-2.5/tutorials/hortonworks/hortonworks-sandbox-hdp2.5-guide/start_sandbox.sh ## start sandbox bash start_sandbox.sh ## configure to start at boot echo "bash /root/start_sandbox.sh" >> /etc/rc.local ## Print the URL for accessing the Sandbox echo -e "##\nAccess the Sandbox at:\nhttp://$(curl -sS4 icanhazip.com):8888\n##" ... View more

Prerequisites: Launch Sandbox on Azure VM Size: Minimum of A4 or A5 A Twitter App You'll use the API credentials The "Application Details" don't matter Prepare the Sandbox Connect to SSH & Ambari Connect to the Sandbox using SSH or web console: http://<<ip>>:4200/ Become root: sudo su - Reset the Ambari password: ambari-admin-password-reset Login to Ambari: http://<<ip>>:8080 User: admin Before moving to the next steps, ensure all services on the left are started (green) or in maintenance mode (black). Install NiFi In Ambari, Click "Actions" (bottom left) -> Add Service Choose NiFi and continue through the dialogs. You shouldn't need to change anything NiFi should now be accessible at http:<<ip>>:9090/nifi/ Tune Sandbox The Sandbox is tuned to run on minimal hardware. We need to update the Hive, Tez & YARN configuration for our use case. This could take up to 15 minutes to complete: bash <(curl -sSL https://git.io/vVRPs) Solr & Banana Solr enables the ability to search across large corpuses of information through specialized indexing techniques. Banana is a dashboard visualization tool for Solr. Download the Banana Dashboard curl -L https://git.io/vVRP3 -o /opt/hostname-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/default.json Update Solr to support Twitter's timestamp format curl -L https://git.io/vVRPz -o /opt/hostname-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml Start Solr JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64 /opt/hostname-hdpsearch/solr/bin/solr start -c -z localhost:2181 Create Solr collection for tweets /opt/hostname-hdpsearch/solr/bin/solr create -c tweets -d data_driven_schema_configs -s 1 -rf 1 ... View more