Newer version available
This article describes how to deploy HDP 2.5 version of demokit. While it can still be used, there is now a newer version of demokit available here that leverages combination of HDP 2.6/HDF 3.0
Overview
Partner demo kit is built and maintained by the Hortonworks Partner Solutions team. The purpose of the demo kit is to enable the partners to:
- Quickly bring up a HDP environments with pre-built demos
- Leverage available demos to understand the capabilities of the platform
- Use the demos as part of business conversation to demonstrate the art of possible
The remainder of this article provides a short description of the 3 demos packaged within the demo kit and step by step instruction on:
- How to launch the demo kit on AWS or on private cloud
- How to execute the demos provided with the demo kit
Pre-requisites
- If using AWS, you must already have created your Amazon Web Services account. Sample steps for doing this can be found here. If you have an AWS promo code, you can apply it to your account using the steps here.
- For running the sentiment demo, you must have created a Twitter application using your Twitter account and generated consumer keys/secrets. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial.
Notes
- Note that the partner demo kit is not a formally supported offering.
- In case of questions, see ‘Questions?” section at the end of this article.
Slides
- Slides for the demo kit are available here
Webinar
- Webinar recording about the demo kit available here
Packaged Demos
The demo kit comes with 3 demos. The slide for these are available here :
- 1. IOT demo
- Purpose: IOT demo showcases how a logistic company uses the Hortonworks Connected Data Platform to monitor its fleet in real time to mitigate driving infractions
- Use case setup:
- Sensor devices from trucks capture events of the trucks and actions of the drivers.
- Some of these driver events are dangerous "events” such as: Lane Departure, Unsafe following distance, Unsafe tail distance
- The Business Requirement is to stream these events in, filter on violations and do real-time alerting when “lots” of erratic behavior is detected for a given driver over a short period of time.
- Over time, users would like to do advanced analytics on the full archive of historical events generated by the trucks to:
- Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc)
- Build an AI model to make predictions when violations will occur
- 2. Sentiment demo
- Purpose: Sentiment demo showcases how a retail company can use the Hortonworks Connected Data Platform to visualize and analyze social media data related to their products
- Use case setup:
- The Business Requirement is to capture, process and analyze flow of tweets to understand the social sentiments for their products
- 3. Advanced analytics demo
- Purpose: Advanced analytics demo showcases how an insurance company can use the Hortonworks Connected Data Platform to visualize and make predictions on earthquake data using Apache Spark’s machine learning libraries
- Use case setup:
- The Business Requirement is to be able to perform advanced analytics on world wide earthquake data to predict where large earthquakes will happen so the business can accordingly modify insurance premiums
Launching the Demo Kit
Option 1: Launching the Demo Kit on AWS using AMI
- 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab.
- 2. Select the Demo Kit AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next
Note: if you choose a smaller instance type from the above recommendation, not all services may come up
- 3. Configure Instance Details: leave the defaults and click ‘Next’
- 4. Add storage: keep the default of 500 GB and click ‘Next’
- 5. Optionally, add a name or any other tags you like. Then click ‘Next’
- 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For production usage, a more restrictive security group policy is strongly encouraged. As an instance only allow traffic from your company’s IP range. Then click ‘Review and Launch’
- 7. Review your settings and click Launch
- 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’
- 9. Click the shown link under ‘Your instances are now launching’
- 10. This opens the EC2 dashboard that shows the details of your launched instance
- 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster) and the ‘Owner’ id (which will be the default password). If the ‘Public IP’ is blank, wait 1-2 minutes for this to be populated
- 12. After about 20 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as admin user using your ‘Owner’ id as password (you can find your owner id in instance details page as highlighted above)
- 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done.
- 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up.
- 15. At this point you can follow the demo instructions.
Troubleshooting
- If any service does not come up for some reason, you can use Ambari to retry by clicking: ‘Actions’ > ‘Start all’.
- In case of multiple failures when starting services, use the EC2 dashboard to double check that the correct instance type was used. Insufficient resources can cause services to not start up successfully
- It is not required to connect via SSH to your instance. But you can do this using the key pair you created/selected earlier by following the standard instructions on AWS website. Make sure the user you login as is ec2-user
- A log file of the automated startup of HDP services is available under: /var/log/hdp_startup.log
- Logs of individual HDP components can be found under /var/log/<component name> or can be accessed using Logsearch UI available at http://<PUBLIC IP>:61888 (login with same credentials as Ambari)
Stopping/Terminating demo kit
- Once you are done with demo kit, we recommend bringing it down to avoid incurring any unnecessary charges. To do this, follow below:
- First, stop the cluster services using Ambari by clicking: ‘Actions’ > ‘Stop all’.
- Then pick from one of the two options:
- a) Terminate the instance: If you do not want to incur any further charges from AWS, terminate the VM instance from the same ‘EC2 dashboard’ that displayed the instance details. Note that this will destroy the VM, so the next time you wish to use demo kit, you will need to follow the same steps outlined in above section ‘Launching the Demo Kit’
- b) Stop the instance: if you want to bring down your VM instance but keep it around so you can start it back up in the future, stop the VM instance from the EC2 dashboard. Note that this option will preserve any customizations you make to the VM but you will incur AWS charges by choosing for this option.
- More details on stop vs terminate operations can be found on AWS website here and here
Option 2: Installing the Demo Kit on other setups
You can also install Demo Kit on other public or private clouds using the provided automated script.
- 1. Launch a CentOS/RHEL 6 or 7 instance on any cloud of your choice with at least 4 cores and 32 GB RAM. Make sure the instance has access to internet.
- Warning: Do NOT run this script on an instance where Ambari or HDP has already been installed (including HDP sandbox)
- 2. SSH into the instance and run the below commands
export host_count=1 export stack=HDPDEMO export ambari_password=BadPass#1 ## change password as needed curl -sSL https://gist.github.com/abajwa-hw/3f2e211d252bba6cad6a6735f78a4a93/raw | sudo -E sh
- 3. This will install Ambari server and agents. After 5-10 min, you should get a message saying the blueprint was deployed which means cluster install has started. At this point you can login to Ambari UI on port 8080 (using user: admin and whatever password you specified above) and monitor the cluster install/startup
Demo Execution Steps
IOT Demo
Video recording of the IOT demo
- Recording of demo provided here
- PPT and PDF versions of the slides also available
- Recording of setup instructions for demo provided here
IOT Demo setup instructions
- Part 1: We will be using the IOT trucking web application to deploy a Storm topology. Then we will use Nifi to push simulated trucking events into Kafka where they will be pulled by Storm for windowing analysis, before being pushed out to a web app and HBase
- In Ambari, open 'IotDemo UI' using quicklink:
- In IotDemo UI, click "Deploy the Storm Topology"
- After 30-60 seconds, the topology will be deployed. Confirm using the Storm View in Ambari:
- Click "Truck Monitoring Application" link in 'IotDemo UI' to open the monitoring app showing an empty map.
- Click 'Nifi Data Flow' in In IotDemo UI to launch Nifi and then double click on 'Iot Trucking demo' processor group. The flow should already be started so no action needed.
- In Ambari, click "Generate Events" to simulate 50 events (this can be configured)
- Switch back to "Truck Monitoring Application" in IotDemo UI and after 30s the trucking events will appear on screen
- Examine the Storm topology using Storm View in Ambari
- Part 2: Next, you can run through the Trucking data Zeppelin notebook to do advanced analytics on the full archive of historical events generated by the trucks to
- Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc)
- Build an AI model to make predictions when violations will occur
- Login to Zeppelin interface using the steps provided under the below section: ‘Advanced Analytics Demo setup instructions’
- Find and open the ‘Trucking Data Analysis’ notebook by filtering for ‘truck’
- On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults.
- Execute the code cells one by one, by clicking the 'Play' (triangular) button on top right of each cell. Alternatively you can just highlight a cell then press Shift-Enter
- You can tell that the status of the cell is RUNNING by the label on the top right of the cell. Note that the first invocation of a cell that runs Spark takes 30-60 seconds as the Spark Application Master is launched on YARN. If desired, you can monitor this using YARN’s Resource Manager UI and Spark UI (for detailed steps, see the below section ‘Advanced Analytics Demo setup instructions’)
Sentiment Demo
Video recording of the Sentiment demo
Sentiment Demo setup instructions
- Open Nifi UI using Quicklinks in Ambari
- Doubleclick "Twitter Dashboard" to open this process group:
- Right click "Grab Garden Hose" > Properties and enter your Twitter Consumer key/secret and Access token/secret. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Optionally change the 'Terms to filter on' as desired. Once complete, start the flow.
- Use Banana UI quicklink from Ambari to open Twitter dashboard
- An empty dashboard will initially appear. After a minute, you should start seeing charts appear
- Use Hive view in Ambari run SQL queries on tweet data
Advanced Analytics Demo
Video recording of Advanced Analytics demo
Advanced Analytics Demo setup instructions
- Open Zeppelin UI via Quicklink
- Login as admin. Password is same as Ambari password
- A directory structure containing a number of demo notebooks will appear.
- Find the earthquake demo notebook by filtering for ‘earthquake’
- On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults.
- Execute the code cells one by one, by clicking the 'Play' (triangular) button on top right of each cell. Alternatively you can just highlight a cell then press Shift-Enter
- You can tell that the status of the cell is RUNNING by the label on the top right of the cell.
- Note that the first invocation of a cell that runs Spark takes 30-60 seconds as the Spark context is launched. Under the covers it is launching a Spark Application Master on YARN. If desired, you can monitor this using Resource Manager UI which is available through Ambari under Yarn > Quicklink.
- The Spark UI can also be access from Resource Manager UI by clicking on application ID for Zeppelin app and then click on ‘Application Master’ hyperlink under ‘Tracking Url’
- The Spark UI can be used to monitor the running Spark jobs
’
This concludes this article on how to launch the demo kit and access the provided demonstrations
Questions?
In case of questions or issues:
- 1. Search on our Hortonworks Community Connection forum. For example, to find all Demo Kit related posts access this url
- 2. If you were not able to find the solution, please post a new question using the tag “partner-demo-kit” here. Please try to be as descriptive as possible when asking questions by providing:
- Detailed description of problem
- Steps to reproduce problem
- Environment details e.g.
- Instance type used was m4.2xlarge
- Storage used was 500gb
- Etc
Announcements
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
