Summary: The release of HDF 3.1 brings about a significant number of improvements in HDF: Apache Nifi 1.5, Kafka 1.0, plus the new NiFi registry. In addition, there were improvements to Storm, Streaming Analytics Manager, Schema Registry components. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy HDF clusters to both either single node or development/demo environments in 5 easy steps. To quickly setup a single node setup, a prebuilt AMI is available for AWS as well as a script that automates these steps, so you can deploy the cluster in a few commands. Steps for each of the below option are described in this article: A. Single-node prebuilt AMI on AWS B. Single-node fresh install C. Multi-node fresh install A. Single-node prebuilt AMI on AWS: Steps to launch the AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep at least the default of 100 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For production usage, a more restrictive security group policy is strongly encouraged. As an instance only allow traffic from your company’s IP range. Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster). If it is blank, wait 1-2 minutes for this to be populated. Also make note of your AWS Owner Id (which will be the initial password to login) 12. After 5-10 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as user:admin and pass:your AWS Owner Id (see previous step) 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. (Optional) You can also monitor the startup using the log as below: Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> Tail the startup log: tail -f /var/log/hdp_startup.log Once you see “cluster is ready!” you can proceed 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. Other related AMIs HDP 2.6.4 vanilla AMI (ami-764d4516): Hortonworks HDP 2.6.4 single node cluster running Hive/Spark/Druid/Superset installed via Ambari. Built Feb 18 2018 using HDP 2.6.4.0-91 / Ambari 2.6.1.3-3. Ambari password is your AWS ownerid HDP 2.6.4 including NiFi and NiFi registry from HDF 3.1 (ami-e1a0a981): HDP 2.6.4 plus NiFi 1.5 and Nifi Registry - Ambari admin password is StrongPassword. Built Feb 17 2018 HDP 2.6 plus HDF 3.0 and IOT trucking demo reference app. Details here Note: Above AMIs are available on US West (N. California) region of AWS B. Single-node HDF install: Launch a fresh CentOS/RHEL 7 instance with 4+cpu and 16GB+ RAM and run below. Do not try to install HDF on a env where Ambari or HDP are already installed (e.g. HDP sandbox or HDP cluster) export host_count=1 curl -sSL https://gist.github.com/abajwa-hw/b7c027d9eea9fbd2a2319a21a955df1f/raw | sudo -E sh Once launched, the script will install Ambari and use it to deploy HDF cluster Note: this script can also be used to install multi-node clusters after step #1 below is complete i.e. after the agents on non-AmabriServer nodes are installed and registered Other related scripts 1. Automation to setup HDP 2.6.x plus NiFi from HDF 3.1 export host_count=1 curl -sSL https://gist.github.com/abajwa-hw/bbe2bdd1ed6a0f738a90dd4e07480e3b/raw | sudo -E sh C. Multi-node HDF install: 0. Launch your RHEL/CentOS 7 instances where you wish to install HDF. In this example, we will use 4 m4.xlarge instances. Select an instance where ambari-server should run (e.g. node1) 1. After choosing a host where you would like Ambari-server to run, first let's prepare the other hosts. Run below on all hosts where Ambari-server will not be running (e.g. node2-4). This will run pre-requisite steps, install Ambari-agents and point them to Ambari-server host: export ambari_server=<FQDN of host where ambari-server will be installed>; #replace this export install_ambari_server=false export ambari_version=2.6.1.0 curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ; 2. Run remaining steps on host where Ambari-server is to be installed (e.g. node1). The below commands run pre-reqs and install Ambari-server export db_password="StrongPassword" # MySQL password export nifi_password="StrongPassword" # NiFi password - must be at least 10 chars export cluster_name="HDF" # cluster name export ambari_services="ZOOKEEPER STREAMLINE NIFI KAFKA STORM REGISTRY NIFI_REGISTRY AMBARI_METRICS" #choose services export hdf_ambari_mpack_url="http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.0.0-564.tar.gz" export ambari_version=2.6.1.0 #install bootstrap yum install -y git python-argparse cd /tmp git clone https://github.com/seanorama/ambari-bootstrap.git #Runs pre-reqs and install ambari-server export install_ambari_server=true curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ; 3. On the same node, install MySQL and create databases and users for Schema Registry and SAM sudo yum localinstall -y https://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm sudo yum install -y epel-release mysql-connector-java* mysql-community-server # MySQL Setup to keep the new services separate from the originals echo Database setup... sudo systemctl enable mysqld.service sudo systemctl start mysqld.service #extract system generated Mysql password oldpass=$( grep 'temporary.*root@localhost' /var/log/mysqld.log | tail -n 1 | sed 's/.*root@localhost: //' ) #create sql file that # 1. reset Mysql password to temp value and create druid/superset/registry/streamline schemas and users # 2. sets passwords for druid/superset/registry/streamline users to ${db_password} cat < mysql-setup.sql ALTER USER 'root'@'localhost' IDENTIFIED BY 'Secur1ty!'; uninstall plugin validate_password; CREATE DATABASE registry DEFAULT CHARACTER SET utf8; CREATE DATABASE streamline DEFAULT CHARACTER SET utf8; CREATE USER 'registry'@'%' IDENTIFIED BY '${db_password}'; CREATE USER 'streamline'@'%' IDENTIFIED BY '${db_password}'; GRANT ALL PRIVILEGES ON registry.* TO 'registry'@'%' WITH GRANT OPTION ; GRANT ALL PRIVILEGES ON streamline.* TO 'streamline'@'%' WITH GRANT OPTION ; commit; EOF #execute sql file mysql -h localhost -u root -p"$oldpass" --connect-expired-password < mysql-setup.sql #change Mysql password to StrongPassword mysqladmin -u root -p'Secur1ty!' password StrongPassword #test password and confirm dbs created mysql -u root -pStrongPassword -e 'show databases;' 4. On the same node, install Mysql connector jar and then HDF mpack. Then restart Ambari so it recognizes HDF stack: sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar sudo ambari-server install-mpack --mpack=${hdf_ambari_mpack_url} --verbose sudo ambari-server restart At this point, if you wanted you could use Ambari install wizard to install HDF you can do that as well. Just open http://<Ambari host IP>:8080 and login and follow the steps in the doc. Otherwise, to proceed with deploying via blueprints follow the remaining steps. 4. On the same node, provide minimum configurations required for install by creating configuration-custom.json. You can add to this to customize any component's property that is exposed by Ambari cd /tmp/ambari-bootstrap/deploy/ tee configuration-custom.json > /dev/null << EOF { "configurations": { "ams-grafana-env": { "metrics_grafana_password": "${db_password}" }, "streamline-common": { "jar.storage.type": "local", "streamline.storage.type": "mysql", "streamline.storage.connector.connectURI": "jdbc:mysql://$(hostname -f):3306/streamline", "registry.url" : "http://localhost:7788/api/v1", "streamline.dashboard.url" : "http://localhost:9089", "streamline.storage.connector.password": "${db_password}" }, "registry-common": { "jar.storage.type": "local", "registry.storage.connector.connectURI": "jdbc:mysql://$(hostname -f):3306/registry", "registry.storage.type": "mysql", "registry.storage.connector.password": "${db_password}" }, "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "${nifi_password}" }, "nifi-ambari-config": { "nifi.security.encrypt.configuration.password": "${nifi_password}" } } } EOF 5. Then run below as root to generate a recommended blueprint and deploy the cluster install. Make sure to set host_count to the total number of hosts in your cluster (including Ambari server) sudo su cd /tmp/ambari-bootstrap/deploy/ export host_count=<Number of total nodes> export ambari_stack_name=HDF export ambari_stack_version=3.1 export ambari_services="ZOOKEEPER STREAMLINE NIFI KAFKA STORM REGISTRY NIFI_REGISTRY AMBARI_METRICS" ./deploy-recommended-cluster.bash You can now login into Ambari at http://<Ambari host IP>:8080 and sit back and watch your HDF cluster get installed! Notes: a) This will only install Nifi on a single node of the cluster by default b) Nifi Certificate Authority (CA) component will be installed by default. This means that if you wanted to, you could enable SSL to be enabled for Nifi out of the box by including a "nifi-ambari-ssl-config" section in the above configuration-custom.json: "nifi-ambari-ssl-config": { "nifi.toolkit.tls.token": "hadoop", "nifi.node.ssl.isenabled": "true", "nifi.security.needClientAuth": "true", "nifi.toolkit.dn.suffix": ", OU=HORTONWORKS", "nifi.initial.admin.identity": "CN=nifiadmin, OU=HORTONWORKS", "content":"<property name='Node Identity 1'>CN=node-1.fqdn, OU=HORTONWORKS</property><property name='Node Identity 2'>CN=node-2.fqdn, OU=HORTONWORKS</property><property name='Node Identity 3'>node-3.fqdn, OU=HORTONWORKS</property>" }, Make sure to replace node-x.fqdn with the FQDN of each node running Nifi c) As part of the install, you can also have an existing Nifi flow deployed by Ambari. First, read in a flow.xml file from existing Nifi system (you can find this in flow.xml.gz). For example, run below to read the flow for the Twitter demo into an env var twitter_flow=$(curl -L https://gist.githubusercontent.com/abajwa-hw/3a3e2b2d9fb239043a38d204c94e609f/raw) Then include a "nifi-ambari-ssl-config" section in the above configuration-custom.json when you run the tee command - to have ambari-bootstrap include the whole flow xml into the generated blueprint: "nifi-flow-env" : { "properties_attributes" : { }, "properties" : { "content" : "${twitter_flow}" } } d) In case you would like to review the generated blueprint before it gets deployed, just set the below variable as well: export deploy=false .... The blueprint will be created under /tmp/ambari-bootstrap*/deploy/tempdir*/blueprint.json Sample blueprint Sample generated blueprint for 4 node cluster is provided for reference here: { "Blueprints": { "stack_name": "HDF", "stack_version": "3.1" }, "host_groups": [ { "name": "host-group-3", "components": [ { "name": "NIFI_MASTER" }, { "name": "DRPC_SERVER" }, { "name": "METRICS_GRAFANA" }, { "name": "KAFKA_BROKER" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "STREAMLINE_SERVER" }, { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "NIMBUS" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "NIFI_REGISTRY_MASTER" }, { "name": "REGISTRY_SERVER" }, { "name": "STORM_UI_SERVER" } ] }, { "name": "host-group-2", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "ZOOKEEPER_SERVER" } ] }, { "name": "host-group-1", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "NIFI_CA" } ] }, { "name": "host-group-4", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "METRICS_COLLECTOR" }, { "name": "ZOOKEEPER_SERVER" } ] } ], "configurations": [ { "nifi-ambari-config": { "nifi.security.encrypt.configuration.password": "StrongPassword" } }, { "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "StrongPassword" } }, { "ams-hbase-env": { "hbase_master_heapsize": "512", "hbase_regionserver_heapsize": "768", "hbase_master_xmn_size": "192" } }, { "nifi-logsearch-conf": {} }, { "storm-site": { "topology.metrics.consumer.register": "[{\"class\": \"org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink\", \"parallelism.hint\": 1, \"whitelist\": [\"kafkaOffset\\..+/\", \"__complete-latency\", \"__process-latency\", \"__execute-latency\", \"__receive\\.population$\", \"__sendqueue\\.population$\", \"__execute-count\", \"__emit-count\", \"__ack-count\", \"__fail-count\", \"memory/heap\\.usedBytes$\", \"memory/nonHeap\\.usedBytes$\", \"GC/.+\\.count$\", \"GC/.+\\.timeMs$\"]}]", "metrics.reporter.register": "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter", "storm.cluster.metrics.consumer.register": "[{\"class\": \"org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter\"}]" } }, { "registry-common": { "registry.storage.connector.connectURI": "jdbc:mysql://ip-172-31-21-233.us-west-1.compute.internal:3306/registry", "registry.storage.type": "mysql", "jar.storage.type": "local", "registry.storage.connector.password": "StrongPassword" } }, { "registry-logsearch-conf": {} }, { "streamline-common": { "streamline.storage.type": "mysql", "jar.storage.type": "local", "streamline.storage.connector.connectURI": "jdbc:mysql://ip-172-31-21-233.us-west-1.compute.internal:3306/streamline", "streamline.dashboard.url": "http://localhost:9089", "registry.url": "http://localhost:7788/api/v1", "streamline.storage.connector.password": "StrongPassword" } }, { "ams-hbase-site": { "hbase.regionserver.global.memstore.upperLimit": "0.35", "hbase.regionserver.global.memstore.lowerLimit": "0.3", "hbase.tmp.dir": "/var/lib/ambari-metrics-collector/hbase-tmp", "hbase.hregion.memstore.flush.size": "134217728", "hfile.block.cache.size": "0.3", "hbase.rootdir": "file:///var/lib/ambari-metrics-collector/hbase", "hbase.cluster.distributed": "false", "phoenix.coprocessor.maxMetaDataCacheSize": "20480000", "hbase.zookeeper.property.clientPort": "61181" } }, { "ams-env": { "metrics_collector_heapsize": "512" } }, { "kafka-log4j": {} }, { "ams-site": { "timeline.metrics.service.webapp.address": "localhost:6188", "timeline.metrics.cluster.aggregate.splitpoints": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetFetch.98percentile", "timeline.metrics.host.aggregate.splitpoints": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetFetch.98percentile", "timeline.metrics.host.aggregator.ttl": "86400", "timeline.metrics.service.handler.thread.count": "20", "timeline.metrics.service.watcher.disabled": "false" } }, { "kafka-broker": { "kafka.metrics.reporters": "org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter" } }, { "ams-grafana-env": { "metrics_grafana_password": "StrongPassword" } }, { "streamline-logsearch-conf": {} } ] } Sample cluster.json for 4 node cluster: { "blueprint": "recommended", "default_password": "hadoop", "host_groups": [ { "hosts": [ { "fqdn": "ip-172-xx-xx-x3.us-west-1.compute.internal" } ], "name": "host-group-3" }, { "hosts": [ { "fqdn": "ip-172-xx-xx-x2.us-west-1.compute.internal" } ], "name": "host-group-2" }, { "hosts": [ { "fqdn": "ip-172-xx-xx-x4.us-west-1.compute.internal" } ], "name": "host-group-4" }, { "hosts": [ { "fqdn": "ip-172-xx-xx-x1.us-west-1.compute.internal" } ], "name": "host-group-1" } ] } What next? Now that your cluster is up, you can explore what Nifi's Ambari integration means: https://community.hortonworks.com/articles/57980/hdf-20-apache-nifi-integration-with-apache-ambarir.html Next, you can enable SSL for Nifi: https://community.hortonworks.com/articles/58009/hdf-20-enable-ssl-for-apache-nifi-from-ambari.html ... View more

Overview Partner demo kit is built and maintained by the Hortonworks Partner Solutions team. The purpose of the demo kit is to enable the partners to: Quickly bring up a HDP environments with pre-built demos Leverage available demos to understand the capabilities of the platform Use the demos as part of business conversation to demonstrate the art of possible The remainder of this article provides a short description of the 3 demos packaged within the demo kit and step by step instruction on: How to launch the demo kit on AWS or on private cloud How to execute the demos provided with the demo kit Other Versions The Security/Governance Demo kit for HDP 2.6 can be found here The previous version of demo kit (for HDP 2.5) can be found here Pre-requisites When using AWS, you must already have created your Amazon Web Services account. Sample steps for doing this can be found here. If you have an AWS promo code, you can apply it to your account using the steps here. For running the sentiment demo, you must have created a Twitter application using your Twitter account and generated consumer keys/secrets. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Notes Note that the partner demo kit is not a formally supported offering. In case of questions, see ‘Questions?” section at the end of this article. Slides Slides for demo kit are available here Packaged Demos The demo kit comes with 3 demos: 1. IOT demo Purpose: IOT demo showcases how a logistic company uses the Hortonworks Connected Data Platform to monitor its fleet in real time to mitigate driving infractions Use case setup: Sensor devices from trucks capture events of the trucks and actions of the drivers. Some of these driver events are dangerous "events” such as: Lane Departure, Unsafe following distance, Unsafe tail distance The Business Requirement is to stream these events in, filter on violations and do real-time alerting when “lots” of erratic behavior is detected for a given driver over a short period of time. Over time, users would like to do advanced analytics on the full archive of historical events generated by the trucks to: Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc) Build an AI model to make predictions when violations will occur Technologies used: Apache Nifi, Kafka, Storm, Streaming Analytics Manager, Schema Registry, HBase, Spark, Zeppelin More details available here and here 2. Sentiment demo Purpose: Sentiment demo showcases how a retail company can use the Hortonworks Connected Data Platform to visualize and analyze social media data related to their products Use case setup: The Business Requirement is to capture, process and analyze flow of tweets to understand the social sentiments for their products Technologies used: Apache Nifi, Solr, HDFS More details available here and here 3. Advanced analytics demo Purpose: Advanced analytics demo showcases how an insurance company can use the Hortonworks Connected Data Platform to visualize and make predictions on earthquake data using Apache Spark’s machine learning libraries Use case setup: The Business Requirement is to be able to perform advanced analytics on world wide earthquake data to predict where large earthquakes will happen so the business can accordingly modify insurance premiums Technologies used: Apache Spark, Zeppelin More details here Option #1: Installing the Demo Kit on your own setup You can install Demo Kit on other public or private clouds using the provided automated script. With this option you would launch a CentOS/RHEL 7 VM of the right size on any cloud of your choice (as long as it has access to public internet), and use provided script to install single node HDP and install the demo. For more details see README here. Setup ETA is 1 hour Option #2: Launching the Demo Kit AMI on AWS You can use this option to launch a prebuilt image of single node HDP (including the demo) on AWS cloud. Setup ETA is 15min Steps to launch the AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep the default of 500 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For long running instances (i.e. anything beyond an hour), a more restrictive security group policy is strongly encouraged (for example: only allow traffic from your company’s IP range). Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster) . If it is blank, wait 1-2 minutes for this to be populated 12. After 5-10 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as admin user using StrongPassword as password 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. (Optional) You can also monitor the startup using the log as below: Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> Tail the startup log: tail -f /var/log/hdp_startup.log Once you see “cluster is ready!” you can proceed 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. If any services fail to start, use the Actions > Start All button to start 15. At this point you can follow the demo instructions. Troubleshooting If any service does not come up for some reason, you can use Ambari to retry by clicking: ‘Actions’ > ‘Start all’. In case of multiple failures when starting services, use the EC2 dashboard to double check that the correct instance type was used. Insufficient resources can cause services to not start up successfully It is not required to connect via SSH to your instance. But you can do this using the key pair you created/selected earlier by following the standard instructions on AWS website. Make sure the user you login as is centos A log file of the automated startup of HDP services is available under: /var/log/hdp_startup.log Stopping/Terminating demo kit Once you are done with demo kit, we recommend bringing it down to avoid incurring any unnecessary charges. To do this, follow below: First, stop the cluster services using Ambari by clicking: ‘Actions’ > ‘Stop all’. Then pick from one of the two options: a) Terminate the instance: If you do not want to incur any further charges from AWS, terminate the VM instance from the same ‘EC2 dashboard’ that displayed the instance details. Note that this will destroy the VM, so the next time you wish to use demo kit, you will need to follow the same steps outlined in above section ‘Launching the Demo Kit’ b) Stop the instance: if you want to bring down your VM instance but keep it around so you can start it back up in the future, stop the VM instance from the EC2 dashboard. Note that this option will preserve any customizations you make to the VM but you will incur AWS charges by choosing for this option. More details on stop vs terminate operations can be found on AWS website here and here Demo Execution Steps IOT Demo Video recording of the IOT demo Recording of demo provided here (high level) and here (deeper level) PPT and PDF versions of the slides also available IOT Demo setup instructions Sequence to walk through the IOT trucking demo: Events simulator Schema Registry UI NiFi flow SAM Application view Storm Monitoring view Superset Dashboard Superset Slice creation Zeppelin notebook Detailed steps for IOT trucking demo walk through (Optional): Check that events are being simulated. This step is optional because we will also check this from NiFi UI Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> sudo su - To check events being simulated you can either verify the simulator process is running or monitor the simulator log: ps -ef | grep stream-simulator tail -f /tmp/whoville/data_simulator/simulator.log If simulator is not running, you can invoke it by running below from SSH sessioncd /tmp/whoville/data_simulator/sudo ./runDataLoader.sh In case you need to kill the simulator use the ps command above to find the process id and then kill it Next, we will open the web UIs of a number of components that are part of the demo using the Ambari Quicklinks. For example, for Schema Registry here is how to access the Quicklink: Open Schema Registry using Quicklink in Ambari and check 4 schemas below are listed Open NiFi using Quicklink in Ambari, check that “IOT trucking demo” process group is started Double click on the “IOT trucking demo” box to see the details of the flow. The counters should show that simulated events are flowing through the NiFi flow. You can refresh the UI to see this: Open Storm Monitoring view (under Ambari views), and check the topology is live Open SAM using Quicklink in Ambari, check the application is deployed Double click on the application to see more details. You should see that the Emitted and Transferred fields are non-zero (assuming the simulator has been been running for a few min) Open Druid Console using Quicklink in Ambari, check the two datasets are present Open Druid Superset using Quicklink in Ambari and login using admin/StrongPassword There should be one entry under Dashboards. Click it to open the prebuilt dashboard. The prebuilt dashboard will open. You can periodically click the refresh button to see new data arriving. Datasets can take 2-6 mins for new events to appear in Druid The first few slices (i.e graphs) provide monitoring related information (e.g. how many violations? Who are the violators? etc). The last 3 slices provide information about the predictions made by the model (i.e. which routes are predicted to have most violations? Which drivers are predicted to have violations) You can also create other slices and add them to the dashboard using the steps here Optionally you can also demonstrate how a data scientist would use archived truck events to build a model to predict violations. Note, to limit amount of resources needed to run the AMI, Spark/Hive has not been installed so you will not be able to actually run the notebook. The previous version of demokit HDP sandbox has these set up so that can be used if you want to actually execute the steps in the notebook. To walk through the trucking events analysis notebook, first open Zeppelin UI using the Quicklink from Ambari: Login as admin/admin Under Notebook section, use search text field to search for “Trucking data analysis” notebook using Zeppelin search: Click Save on the interpreter binding Walk through the notebook to show how data scientist can use SparkSQL to visualize data to help understand what features should be included in the model Finally you can show that once the important features are known, a model can be built to predict violations (in this case, using Logistical Regression) Stopping/Starting the simulator To stop the simulator, use below command to find its process id and then use kill command to kill it: ps -ef | grep stream-simulator kill <process_id> To start it back up, run below:cd /tmp/whoville/data_simulator/sudo ./runDataLoader.sh Sentiment Demo Video recording of the Sentiment demo Recording of setup instructions for demo provided here Sentiment Demo setup instructions Open Nifi UI using Quicklinks in Ambari Doubleclick "Twitter Dashboard" to open this process group: Right click "Grab Garden Hose" > Properties and enter your Twitter Consumer key/secret and Access token/secret. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Optionally change the 'Terms to filter on' as desired. Once complete, start the flow. Use Banana UI quicklink from Ambari to open Twitter dashboard An empty dashboard will initially appear. After a minute, you should start seeing charts appear Advanced Analytics Demo Video recording of Advanced Analytics demo Video recording provided here Advanced Analytics Demo setup instructions Open Zeppelin UI via Quicklink Login as admin. Password is same as Ambari password A directory structure containing a number of demo notebooks will appear. Find the earthquake demo notebook by filtering for ‘earthquake’ On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults. Now you can walk through the notebook and show the visualizations and process of building the model. Note, to limit amount of resources needed to run the AMI, Spark/Hive has not been installed so you will not be able to actually run the notebook. The previous version of demokit or HDP sandbox has the notebook set up so that can be used if you want to actually execute the steps in the notebook. This concludes this article on how to launch the demo kit and access the provided demonstrations Questions? In case of questions or issues: 1. Search on our Hortonworks Community Connection forum. For example, to find all Demo Kit related posts access this url 2. If you were not able to find the solution, please post a new question using the tag “partner-demo-kit” here. Please try to be as descriptive as possible when asking questions by providing: Detailed description of problem Steps to reproduce problem Environment details e.g. Instance type used was m4.2xlarge Storage used was 500gb Etc Relevant log file snippets ... View more

Newer version available This article describes how to deploy HDP 2.5 version of demokit. While it can still be used, there is now a newer version of demokit available here that leverages combination of HDP 2.6/HDF 3.0 Overview Partner demo kit is built and maintained by the Hortonworks Partner Solutions team. The purpose of the demo kit is to enable the partners to: Quickly bring up a HDP environments with pre-built demos Leverage available demos to understand the capabilities of the platform Use the demos as part of business conversation to demonstrate the art of possible The remainder of this article provides a short description of the 3 demos packaged within the demo kit and step by step instruction on: How to launch the demo kit on AWS or on private cloud How to execute the demos provided with the demo kit Pre-requisites If using AWS, you must already have created your Amazon Web Services account. Sample steps for doing this can be found here. If you have an AWS promo code, you can apply it to your account using the steps here. For running the sentiment demo, you must have created a Twitter application using your Twitter account and generated consumer keys/secrets. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Notes Note that the partner demo kit is not a formally supported offering. In case of questions, see ‘Questions?” section at the end of this article. Slides Slides for the demo kit are available here Webinar Webinar recording about the demo kit available here Packaged Demos The demo kit comes with 3 demos. The slide for these are available here : 1. IOT demo Purpose: IOT demo showcases how a logistic company uses the Hortonworks Connected Data Platform to monitor its fleet in real time to mitigate driving infractions Use case setup: Sensor devices from trucks capture events of the trucks and actions of the drivers. Some of these driver events are dangerous "events” such as: Lane Departure, Unsafe following distance, Unsafe tail distance The Business Requirement is to stream these events in, filter on violations and do real-time alerting when “lots” of erratic behavior is detected for a given driver over a short period of time. Over time, users would like to do advanced analytics on the full archive of historical events generated by the trucks to: Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc) Build an AI model to make predictions when violations will occur Technologies used: Apache Nifi, Kafka, Storm, HBase, Spark, Zeppelin More details available here and here 2. Sentiment demo Purpose: Sentiment demo showcases how a retail company can use the Hortonworks Connected Data Platform to visualize and analyze social media data related to their products Use case setup: The Business Requirement is to capture, process and analyze flow of tweets to understand the social sentiments for their products Technologies used: Apache Nifi, Solr, Hive, HDFS More details available here and here 3. Advanced analytics demo Purpose: Advanced analytics demo showcases how an insurance company can use the Hortonworks Connected Data Platform to visualize and make predictions on earthquake data using Apache Spark’s machine learning libraries Use case setup: The Business Requirement is to be able to perform advanced analytics on world wide earthquake data to predict where large earthquakes will happen so the business can accordingly modify insurance premiums Technologies used: Apache Spark, Zeppelin More details here Launching the Demo Kit Option 1: Launching the Demo Kit on AWS using AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the Demo Kit AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep the default of 500 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For production usage, a more restrictive security group policy is strongly encouraged. As an instance only allow traffic from your company’s IP range. Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster) and the ‘Owner’ id (which will be the default password). If the ‘Public IP’ is blank, wait 1-2 minutes for this to be populated 12. After about 20 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as admin user using your ‘Owner’ id as password (you can find your owner id in instance details page as highlighted above) 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. 15. At this point you can follow the demo instructions. Troubleshooting If any service does not come up for some reason, you can use Ambari to retry by clicking: ‘Actions’ > ‘Start all’. In case of multiple failures when starting services, use the EC2 dashboard to double check that the correct instance type was used. Insufficient resources can cause services to not start up successfully It is not required to connect via SSH to your instance. But you can do this using the key pair you created/selected earlier by following the standard instructions on AWS website. Make sure the user you login as is ec2-user A log file of the automated startup of HDP services is available under: /var/log/hdp_startup.log Logs of individual HDP components can be found under /var/log/<component name> or can be accessed using Logsearch UI available at http://<PUBLIC IP>:61888 (login with same credentials as Ambari) Stopping/Terminating demo kit Once you are done with demo kit, we recommend bringing it down to avoid incurring any unnecessary charges. To do this, follow below: First, stop the cluster services using Ambari by clicking: ‘Actions’ > ‘Stop all’. Then pick from one of the two options: a) Terminate the instance: If you do not want to incur any further charges from AWS, terminate the VM instance from the same ‘EC2 dashboard’ that displayed the instance details. Note that this will destroy the VM, so the next time you wish to use demo kit, you will need to follow the same steps outlined in above section ‘Launching the Demo Kit’ b) Stop the instance: if you want to bring down your VM instance but keep it around so you can start it back up in the future, stop the VM instance from the EC2 dashboard. Note that this option will preserve any customizations you make to the VM but you will incur AWS charges by choosing for this option. More details on stop vs terminate operations can be found on AWS website here and here Option 2: Installing the Demo Kit on other setups You can also install Demo Kit on other public or private clouds using the provided automated script. 1. Launch a CentOS/RHEL 6 or 7 instance on any cloud of your choice with at least 4 cores and 32 GB RAM. Make sure the instance has access to internet. Warning: Do NOT run this script on an instance where Ambari or HDP has already been installed (including HDP sandbox) 2. SSH into the instance and run the below commands export host_count=1 export stack=HDPDEMO export ambari_password=BadPass#1 ## change password as needed curl -sSL https://gist.github.com/abajwa-hw/3f2e211d252bba6cad6a6735f78a4a93/raw | sudo -E sh 3. This will install Ambari server and agents. After 5-10 min, you should get a message saying the blueprint was deployed which means cluster install has started. At this point you can login to Ambari UI on port 8080 (using user: admin and whatever password you specified above) and monitor the cluster install/startup Demo Execution Steps IOT Demo Video recording of the IOT demo Recording of demo provided here PPT and PDF versions of the slides also available Recording of setup instructions for demo provided here IOT Demo setup instructions Part 1: We will be using the IOT trucking web application to deploy a Storm topology. Then we will use Nifi to push simulated trucking events into Kafka where they will be pulled by Storm for windowing analysis, before being pushed out to a web app and HBase In Ambari, open 'IotDemo UI' using quicklink: In IotDemo UI, click "Deploy the Storm Topology" After 30-60 seconds, the topology will be deployed. Confirm using the Storm View in Ambari: Click "Truck Monitoring Application" link in 'IotDemo UI' to open the monitoring app showing an empty map. Click 'Nifi Data Flow' in In IotDemo UI to launch Nifi and then double click on 'Iot Trucking demo' processor group. The flow should already be started so no action needed. In Ambari, click "Generate Events" to simulate 50 events (this can be configured) Switch back to "Truck Monitoring Application" in IotDemo UI and after 30s the trucking events will appear on screen Examine the Storm topology using Storm View in Ambari Part 2: Next, you can run through the Trucking data Zeppelin notebook to do advanced analytics on the full archive of historical events generated by the trucks to Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc) Build an AI model to make predictions when violations will occur Login to Zeppelin interface using the steps provided under the below section: ‘Advanced Analytics Demo setup instructions’ Find and open the ‘Trucking Data Analysis’ notebook by filtering for ‘truck’ On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults. Execute the code cells one by one, by clicking the 'Play' (triangular) button on top right of each cell. Alternatively you can just highlight a cell then press Shift-Enter You can tell that the status of the cell is RUNNING by the label on the top right of the cell. Note that the first invocation of a cell that runs Spark takes 30-60 seconds as the Spark Application Master is launched on YARN. If desired, you can monitor this using YARN’s Resource Manager UI and Spark UI (for detailed steps, see the below section ‘Advanced Analytics Demo setup instructions’) Sentiment Demo Video recording of the Sentiment demo Recording of setup instructions for demo provided here Slides available here Sentiment Demo setup instructions Open Nifi UI using Quicklinks in Ambari Doubleclick "Twitter Dashboard" to open this process group: Right click "Grab Garden Hose" > Properties and enter your Twitter Consumer key/secret and Access token/secret. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Optionally change the 'Terms to filter on' as desired. Once complete, start the flow. Use Banana UI quicklink from Ambari to open Twitter dashboard An empty dashboard will initially appear. After a minute, you should start seeing charts appear Use Hive view in Ambari run SQL queries on tweet data Advanced Analytics Demo Video recording of Advanced Analytics demo Video recording provided here Slides provided here Advanced Analytics Demo setup instructions Open Zeppelin UI via Quicklink Login as admin. Password is same as Ambari password A directory structure containing a number of demo notebooks will appear. Find the earthquake demo notebook by filtering for ‘earthquake’ On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults. Execute the code cells one by one, by clicking the 'Play' (triangular) button on top right of each cell. Alternatively you can just highlight a cell then press Shift-Enter You can tell that the status of the cell is RUNNING by the label on the top right of the cell. Note that the first invocation of a cell that runs Spark takes 30-60 seconds as the Spark context is launched. Under the covers it is launching a Spark Application Master on YARN. If desired, you can monitor this using Resource Manager UI which is available through Ambari under Yarn > Quicklink. The Spark UI can also be access from Resource Manager UI by clicking on application ID for Zeppelin app and then click on ‘Application Master’ hyperlink under ‘Tracking Url’ The Spark UI can be used to monitor the running Spark jobs ’ This concludes this article on how to launch the demo kit and access the provided demonstrations Questions? In case of questions or issues: 1. Search on our Hortonworks Community Connection forum. For example, to find all Demo Kit related posts access this url 2. If you were not able to find the solution, please post a new question using the tag “partner-demo-kit” here. Please try to be as descriptive as possible when asking questions by providing: Detailed description of problem Steps to reproduce problem Environment details e.g. Instance type used was m4.2xlarge Storage used was 500gb Etc Relevant log file snippets ... View more

In the previous articles, we showed how to deploy an HDF 2.x/3.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Here we will build on the same cluster and show how to enable kerberos using Active Directory. Summary To achieve this, the high level steps we will follow are: Setup certificate trust for HDF nodes Run Ambari security wizard Create Ranger policy for nifiadmin user Delete certificate Login to Nifi using AD principal credentials Pre-requisites You have correctly setup AD as described here Active Directory setup with domain: CLOUD.HORTONWORKS.COM AD already preconfigured with LDAPS Certificate (.crt) used to enable LDAPS is available OU created where HDF principals will be created hadoop user has permission to write principals to above OU nifiadmin user created in AD (optionally synced over to Ranger) Test to ensure you can access AD over LDAPS using hadoopadmin user succeeds: ldapsearch -H ldaps://sme-security-ad03.cloud.hortonworks.com:636 -D hadoopadmin@cloud.hortonworks.com -w BadPass#1 Steps 1. Setup trust for all HDF nodes using the AD certificate #run on all HDF nodes before running security wizard using AD ad_ip=xx.xx.xx.xx ##replace with IP of your AD cert_url=http://someurl/mycertificate.crt ## replace with location of exported AD certificate echo "${ad_ip} ad01.lab.hortonworks.net ad01" | sudo tee -a /etc/hosts sudo yum -y install openldap-clients ca-certificates #instead of downloading the cert, you could also manually transfer the .cert file to below location sudo curl -sSL "${cert_url}" -o /etc/pki/ca-trust/source/anchors/hortonworks-net.crt sudo update-ca-trust force-enable sudo update-ca-trust extract sudo update-ca-trust check # edit /etc/openldap/ldap.conf to include LDAP url and base sudo tee -a /etc/openldap/ldap.conf > /dev/null << EOF TLS_CACERT /etc/pki/tls/cert.pem URI ldaps://ad01.lab.hortonworks.net ldap://ad01.lab.hortonworks.net BASE dc=cloud,dc=hortonworks,dc=com EOF #test using openssl - should return 0 openssl s_client -connect ad01:636 </dev/null #test using ldapsearch ldapsearch -H ldaps://sme-security-ad03.cloud.hortonworks.com:636 -D nifiadmin@cloud.hortonworks.com -w BadPass#1 2. Run Ambari Security Wizard Launch security wizard via Ambari (under Admin > Kerberos) and enter below: The ‘Configure Kerberos’ page is the only one you will need to update. Enter the below then click Next on all remaining screens. KDC host: FQDN of AD Realm name: CLOUD.HORTONWORKS.COM Kadmin host: FQDN of AD node Admin principal: hadoopadmin@cloud.hortonworks.com Password: BadPass#1 On ‘Configure Identities’ page, users will be shown the option to customize the keytabs/principals for all components: The Nifi ones are under Advanced tab: Click Next to proceed using the default keytab/principal names Click Next to proceed through all remaining steps of the wizard. What’s happening to Nifi under the covers when security wizard runs? a) NiFi principal and keytabs will be automatically be created/distributed across the cluster where needed by Ambari b) Kerberos-related nifi.properties fields will automatically be updated: NiFi.kerberos.service.principal NiFi.kerberos.keytab.location NiFi.kerberos.krb5.file NiFi.kerberos.authentication.expiration c) Login provider will also be switched to kerberos under the covers d) As part of the process, other HDF components were also kerberized including ‘Ambari Infra’ service. This mean that Ranger audits are now being written to kerberized Solr After security wizard completes, NiFi’s kerberos details will appear alongside other components (under Admin > Kerberos). At this point, Kerberos security will be enabled for all components running on the cluster: On a node running Nifi, you can verify the keytab was generated and list its principal # klist -kt /etc/security/keytabs/nifi.service.keytab Keytab name: FILE:/etc/security/keytabs/nifi.service.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM 1 09/28/2016 04:55:08 nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM You can also verify the nifi configs for kerberos were automatically populated: # cat /etc/nifi/conf/nifi.properties | grep kerberos nifi.kerberos.krb5.file=/etc/krb5.conf nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.service.keytab nifi.kerberos.service.principal=nifi/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM nifi.kerberos.spnego.authentication.expiration=12 hours nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/spnego.service.keytab nifi.kerberos.spnego.principal=HTTP/abajwa-hdf-qe-hdfsecured-1.openstacklocal@CLOUD.HORTONWORKS.COM nifi.security.user.login.identity.provider=kerberos-provider You can also verify that the login-identity-provider or Nifi has now been switched to kerberos # tail /etc/nifi/conf/login-identity-providers.xml <provider> <identifier>kerberos-provider</identifier> <class>org.apache.nifi.kerberos.KerberosProvider</class> <property name="Default Realm">HORTONWORKS.COM</property> <property name="Authentication Expiration">12 hours</property> </provider> 3. Login to Nifi UI without certificate Now that kerberos is enabled, lets try to login without using certificate Make sure nifiadmin user exists in Ranger (if you ran Ranger sync earlier this should have been imported already). If not, create the user in Ranger by navigating to below url and entering below http://<Ranger_node>:6080/index.html#!/user/create Create Ranger policy for new user In Ranger, under ‘Access Manager, click ‘HDF-nifi’ Click Edit button on the /* policy we previously added nifiadmin@CLOUD.HORTONWORKS.COM to Add the newly created nifiadmin user to the policy, and click Save Delete previously imported .p12 certificates from your browser e.g. if using Chrome on OSX you can delete previously imported certificates using ‘Keychain Access’ application Restart Chrome and open Nifi UI. It should now display a login page If not, try opening “Incognito Window” Enter username as nifiadmin and the password you set The Nifi UI should open now and you will be logged in as that user You can see who you are logged in as by checking top-right corner of Nifi UI This completes the tutorial. If you made it this far in the series, congratulations! You have successfully: Deployed HDF 2.0 Enabled SSL for Nifi and explored file-based authorization for Nifi Installed Ranger and switched to Ranger-based authorization for Nifi Enabled kerberos for your HDF cluster using Active Directory Logged into Nifi using AD credentials ... View more