Summary:
Automation/AMI to install HDP 2.5.x with Nifi 1.1.0 on any cloud and deploy commonly used demos via Ambari blueprints
Currently supported demos:
- Nifi-Twitter
- IoT (trucking) demo
- Zeppelin notebooks
- Vanilla HDF 2.1 (w/o any demos)
Option 1: Deploy single node instances using AMIs
1. For deploying the above on single node setups on Amazon, AMI images are also available. To launch an instance using one of the AMIs, refer to steps below. A video that shows using these steps to launch the HDP 2.5.3 AMI is available here.
- Login into EC2 dashboard using your credentials
- Change your region to "N. California"
- Click 'Launch instance'
- Choose AMI: search for 081339556850 under Community AMIs (as shown in screenshot), select the desired AMI. For the HDP 2.5.x version of the AMI that has the demos pre-installed, select "HDP 2.5 Demo kit cluster"
- Choose instance type: select m4.2xlarge for HDP AMIs or m4.xlarge for HDF
- Configure instance: leave defaults
- Add storage: 100gb or larger (500gb preferred)
- Tag: name your instance and add any tags you like
- Configure Security Group: choose security group that opens all the ports (e.g. sg-1c53d279summit2015) or create new
- While deploying choose an SSH key you have the .pem file for or create new
2. Once the instance comes up and Ambari server/agent are fully up, it will automatically start the services. You can monitor this by connecting to your instance via SSH as ec2-user and tailing /var/log/hdp_startup.log
3. Once the service start call was made, you can login to Ambari UI (port 8080) to monitor progress. Note: if Ambari is not accessible make sure a) the security group you used has a policy for 8080 b) you waited enough time for Ambari to come up.
The password for 'admin' user of Ambari and Zeppelin is defaulted to your AWS account number. You can look this up using your EC2 dashboard as below
3. So 15-20 min after AWS shows the instance came up, you should see a fully started cluster. Note: in case any service does not come up, you can bring it up using 'Service Actions' menu in Ambari
Notes:
- Once the cluster is up, it is recommended that you change the Ambari and Zeppelin admin passwords
- The instance launched is EBS backed - so the VM can be stopped when not in use and restarted when needed. Just make sure to stop all HDP/HDF services via Ambari before stopping the instance via EC2 dashboard.
What gets installed?
- HDP 2.5.x with below vanilla components
- IotDemo demo service - allows users to stop/start Iot Demo, open webUI and generate events
- Demo Ambari service for Solr
- This service will pre-configure Solr/Banana for Twitter demo
- Demo Ambari service for Nifi 1.1
- The script auto-deploys the specified flow - by default, it deploys the the Twitter flow but this is overridable
- Even though the flow is deployed, you will need to set processors that contain env-specific details e.g. you will need to enter Twitter key/secret in GetTwitter processor
IoT Trucking demo steps
Once the instance is up, you can follow the below steps to start the trucking demo. Video here
- In Ambari, open 'IotDemo UI' using quicklink:
- In IotDemo UI, click "Deploy the Storm Topology"
- After 30-60 seconds, the topology will be deployed. Confirm using the Storm View in Ambari:
- Click "Truck Monitoring Application" link in 'IotDemo UI' to open the monitoring app showing an empty map.
- Click 'Nifi Data Flow' in In IotDemo UI to launch Nifi and then double click on 'Iot Trucking demo' processor group. Then right click on both PublishKafka_0_10 processors > Configure > Properties. Confirm that the 'Kafka Broker' hostname/port is correctly populated. The flow should already be started so no other action needed.
- In Ambari, click "Generate Events" to simulate 50 events (this can be configured)
- Switch back to "Truck Monitoring Application" in IotDemo UI and after 30s the trucking events will appear on screen
- Explore Storm topology using Storm View in Ambari
Nifi Sentiment demo
Next you can follow the below steps to start the Nifi sentiment demo. Video of these steps available here
- Open Nifi UI using Quicklinks in Ambari
- Double click "Twitter Dashboard" to open this process group:
- Right click "Grab Garden Hose" > Properties and enter your Twitter Consumer key/secret and Access token/secret. Optionally change the 'Terms to filter on' as desired. Once complete, start the flow.
- Use Banana UI quicklink from Ambari to open Twitter dashboard
- An empty dashboard will initially appear. After a minute, you should start seeing charts appear
Zeppelin demos
- Open Zeppelin UI via Quicklink
- Login as admin. Password is same as Ambari password
- Demo notebooks will appear. Open the first notebook and walk through each cell.
Option 2: To install HDP (including demos) or HDF using scripts
Pre-reqs:
- One or more freshly installed CentOS/RHEL 6 or 7 VMs on your cloud of choice
- Do not run this script on VMs running an existing HDP cluster or sandbox
- If planning to install ‘IoT Demo’ make sure you allocate enough memory - especially if also deploying other demos
- 16GB or more of RAM is recommended if using single node setup
- The sample script should only be used to create test/demo clusters
- Default password for Ambari and Zeppelin admin users is BadPass#1
- Override by exporting ambari_password prior to running the script
Steps:
1. This step is only needed if installing a multi-node cluster. After choosing a host where you would like Ambari-server to run, first prepare the other hosts. Run this on all hosts where Ambari-server will not be running to run pre-requisite steps, install Ambari-agents and point them to Ambari-server host:
export ambari_server=<FQDN of ambari-server host> curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ;
2. Run remaining steps on host where Ambari-server is to be installed. These run pre-reqs and install Ambari-server and deploy demos requested
a) To install HDP 2.5.x (Ambari 2.4.1/Java 😎 - including Solr/Nifi 1.0.0 via Ambari and deploy a Nifi flow:
export host_count=1 #set to number of nodes in your cluster (including Ambari-server node) export hdp_ver=2.5 export install_nifidemo=true export install_iotdemo=true curl -sSL https://gist.github.com/abajwa-hw/3f2e211d252bba6cad6a6735f78a4a93/raw | sudo -E sh
After 5-10 min, you should get a message saying the blueprint was deployed. At this point you can open Ambari UI (port 8080) and monitor the cluster install
Note: if you installed iotdemo on a multi-node cluster, there maybe some manual steps required (e.g. moving storm jars or setting up latest Storm view). See here for more info: https://github.com/hortonworks-gallery/iotdemo-service/tree/hdp25#post-install-manual-steps
b) To install HDP 2.4 (Ambari 2.4.1/java 😎 - including IoTDemo, plus Solr/Nifi 1.0.0 via Ambari and deploy Nifi Twitter flow run below:
export host_count=1 #set to number of nodes in your cluster (including Ambari-server node) export hdp_ver=2.4 export install_iotdemo=true export install_nifidemo=true curl -sSL https://gist.github.com/abajwa-hw/3f2e211d252bba6cad6a6735f78a4a93/raw | sudo -E sh
c) To install vanilla HDF 2.1 cluster, you can use the script/steps below:
Note this does not install any of the demos, just a vanilla HDF 2.1 cluster
Deployment
After 5-10min, you should get a message saying the blueprint was deployed. At this point you can open Ambari UI (port 8080) and monitor the cluster install. (Note make sure the port was opened). Default password is BadPass#1
What gets installed?
- refer to previous 'What gets installed' section
