Summary: While automating setup of Hortoniabank demo, we needed to automate the task of associating Atlas tags to HDP entities like HDFS, Hive, HBase, Kafka using the names of entities (rather than their guids in Atlas). One option is to use Atlas APIs to find the entity you are looking for using qualifiedName attribute and then use the guid to associates tag to it. For components like Hive that already have Atlas hook, the Atlas entities for Hive tables will automatically be created when the table is created. For these, we have just provided the API calls to associate the tags with the entity. For others like Kafka, HDFS, Hbase etc that do not have an Atlas hook (as of HDP 2.6.x), you will need to create the entity first. For these, we have provided both the API call to create the entity and the call to associate the tags with the entity. Code samples: The below code examples assume the tags have already been created. these can be created either manually via Atlas UI or using the API. Here is a sample Atlas API call to create a basic tag called TEST that does not have any attributes. ${atlas_curl} ${atlas_url}/types \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"enumTypes":[],"structTypes":[],"traitTypes":[{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.TraitType","typeName":"TEST","typeDescription":"TEST","typeVersion":"1.0","attributeDefinitions":[]}],"classTypes":[]}' All the examples operate the same way: find the guid of the entity you are looking for using qualifiedName attribute and then use the guid to associates tag to it. First we setup common vars: atlas_host="atlas.domain.com" cluster_name="datalake" atlas_curl="curl -u admin:admin" atlas_url="http://${atlas_host}:21000/api/atlas" Example 1: Associate tag REFERENCE_DATA (w/o attributes) to Hive table hortoniabank.eu_countries #fetch guid for table hortoniabank.eu_countries@${cluster_name} guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hive_table?attr:qualifiedName=hortoniabank.eu_countries@${cluster_name} | jq '.entity.guid' | tr -d '"') #add REFERENCE_DATA tag ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"REFERENCE_DATA","values":{}}' Example 2: Associate tag DATA_QUALITY (with attribute: score and value: 0.51) to Hive table cost_savings.claim_savings #fetch guid for table cost_savings.claim_savings@${cluster_name} guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hive_table?attr:qualifiedName=cost_savings.claim_savings@${cluster_name} | jq '.entity.guid' | tr -d '"') #add DATA_QUALITY tag with score=0.51 ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"DATA_QUALITY", "values":{"score": "0.51"}}' Example 3: Associate tag FINANCE_PII (with attribute: type and value:finance) to Hive column finance.tax_2015.ssn #fetch guid for finance.tax_2015.ssn guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hive_column?attr:qualifiedName=finance.tax_2015.ssn@${cluster_name} | jq '.entity.guid' | tr -d '"') #add FINANCE_PII tag with type=finance ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"FINANCE_PII", "values":{"type": "finance"}}' Example 4: Create entity for kafka topic PRIVATE and associate with tag SENSITIVE #create entities for kafka topics PRIVATE and associate with SENSITIVE tag ${atlas_curl} ${atlas_url}/v2/entity -X POST -H 'Content-Type: application/json' -d @- <<EOF { "entity":{ "typeName":"kafka_topic", "attributes":{ "description":null, "name":"PRIVATE", "owner":null, "qualifiedName":"PRIVATE@${cluster_name}", "topic":"PRIVATE", "uri":"none" }, "guid":-1 }, "referredEntities":{ }} EOF guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/kafka_topic?attr:qualifiedName=PRIVATE@${cluster_name} | jq '.entity.guid' | tr -d '"') ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"SENSITIVE","values":{}}' Example 5: create entities for Hbase table T_PRIVATE and associate with SENSITIVE tag #create entities for Hbase table T_PRIVATE and associate with SENSITIVE tag ${atlas_curl} ${atlas_url}/v2/entity -X POST -H 'Content-Type: application/json' -d @- <<EOF { "entity":{ "typeName":"hbase_table", "attributes":{ "description":"T_PRIVATE table", "name":"T_PRIVATE", "owner":"hbase", "qualifiedName":"T_PRIVATE@${cluster_name}", "column_families":[ ], "uri":"none" }, "guid":-1 }, "referredEntities":{ }} EOF guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hbase_table?attr:qualifiedName=T_PRIVATE@${cluster_name} | jq '.entity.guid' | tr -d '"') ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"SENSITIVE","values":{}}' Example 6: create entities for HDFS path /banking and associate with BANKING tag #create entities for HDFS path /banking and associate with BANKING tag hdfs_prefix="hdfs://$(hostname -f):8020" hdfs_path="/banking" ${atlas_curl} ${atlas_url}/v2/entity -X POST -H 'Content-Type: application/json' -d @- <<EOF { "entity":{ "typeName":"hdfs_path", "attributes":{ "description":null, "name":"${hdfs_path}", "owner":null, "qualifiedName":"${hdfs_prefix}${hdfs_path}", "clusterName":"${cluster_name}", "path":"${hdfs_prefix}${hdfs_path}" }, "guid":-1 }, "referredEntities":{ }} EOF guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hdfs_path?attr:qualifiedName=${hdfs_prefix}${hdfs_path} | jq '.entity.guid' | tr -d '"') ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"BANKING","values":{}}' ... View more

Summary: The release of HDF 3.1 brings about a significant number of improvements in HDF: Apache Nifi 1.5, Kafka 1.0, plus the new NiFi registry. In addition, there were improvements to Storm, Streaming Analytics Manager, Schema Registry components. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy HDF clusters to both either single node or development/demo environments in 5 easy steps. To quickly setup a single node setup, a prebuilt AMI is available for AWS as well as a script that automates these steps, so you can deploy the cluster in a few commands. Steps for each of the below option are described in this article: A. Single-node prebuilt AMI on AWS B. Single-node fresh install C. Multi-node fresh install A. Single-node prebuilt AMI on AWS: Steps to launch the AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep at least the default of 100 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For production usage, a more restrictive security group policy is strongly encouraged. As an instance only allow traffic from your company’s IP range. Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster). If it is blank, wait 1-2 minutes for this to be populated. Also make note of your AWS Owner Id (which will be the initial password to login) 12. After 5-10 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as user:admin and pass:your AWS Owner Id (see previous step) 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. (Optional) You can also monitor the startup using the log as below: Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> Tail the startup log: tail -f /var/log/hdp_startup.log Once you see “cluster is ready!” you can proceed 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. Other related AMIs HDP 2.6.4 vanilla AMI (ami-764d4516): Hortonworks HDP 2.6.4 single node cluster running Hive/Spark/Druid/Superset installed via Ambari. Built Feb 18 2018 using HDP 2.6.4.0-91 / Ambari 2.6.1.3-3. Ambari password is your AWS ownerid HDP 2.6.4 including NiFi and NiFi registry from HDF 3.1 (ami-e1a0a981): HDP 2.6.4 plus NiFi 1.5 and Nifi Registry - Ambari admin password is StrongPassword. Built Feb 17 2018 HDP 2.6 plus HDF 3.0 and IOT trucking demo reference app. Details here Note: Above AMIs are available on US West (N. California) region of AWS B. Single-node HDF install: Launch a fresh CentOS/RHEL 7 instance with 4+cpu and 16GB+ RAM and run below. Do not try to install HDF on a env where Ambari or HDP are already installed (e.g. HDP sandbox or HDP cluster) export host_count=1 curl -sSL https://gist.github.com/abajwa-hw/b7c027d9eea9fbd2a2319a21a955df1f/raw | sudo -E sh Once launched, the script will install Ambari and use it to deploy HDF cluster Note: this script can also be used to install multi-node clusters after step #1 below is complete i.e. after the agents on non-AmabriServer nodes are installed and registered Other related scripts 1. Automation to setup HDP 2.6.x plus NiFi from HDF 3.1 export host_count=1 curl -sSL https://gist.github.com/abajwa-hw/bbe2bdd1ed6a0f738a90dd4e07480e3b/raw | sudo -E sh C. Multi-node HDF install: 0. Launch your RHEL/CentOS 7 instances where you wish to install HDF. In this example, we will use 4 m4.xlarge instances. Select an instance where ambari-server should run (e.g. node1) 1. After choosing a host where you would like Ambari-server to run, first let's prepare the other hosts. Run below on all hosts where Ambari-server will not be running (e.g. node2-4). This will run pre-requisite steps, install Ambari-agents and point them to Ambari-server host: export ambari_server=<FQDN of host where ambari-server will be installed>; #replace this export install_ambari_server=false export ambari_version=2.6.1.0 curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ; 2. Run remaining steps on host where Ambari-server is to be installed (e.g. node1). The below commands run pre-reqs and install Ambari-server export db_password="StrongPassword" # MySQL password export nifi_password="StrongPassword" # NiFi password - must be at least 10 chars export cluster_name="HDF" # cluster name export ambari_services="ZOOKEEPER STREAMLINE NIFI KAFKA STORM REGISTRY NIFI_REGISTRY AMBARI_METRICS" #choose services export hdf_ambari_mpack_url="http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.0.0-564.tar.gz" export ambari_version=2.6.1.0 #install bootstrap yum install -y git python-argparse cd /tmp git clone https://github.com/seanorama/ambari-bootstrap.git #Runs pre-reqs and install ambari-server export install_ambari_server=true curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ; 3. On the same node, install MySQL and create databases and users for Schema Registry and SAM sudo yum localinstall -y https://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm sudo yum install -y epel-release mysql-connector-java* mysql-community-server # MySQL Setup to keep the new services separate from the originals echo Database setup... sudo systemctl enable mysqld.service sudo systemctl start mysqld.service #extract system generated Mysql password oldpass=$( grep 'temporary.*root@localhost' /var/log/mysqld.log | tail -n 1 | sed 's/.*root@localhost: //' ) #create sql file that # 1. reset Mysql password to temp value and create druid/superset/registry/streamline schemas and users # 2. sets passwords for druid/superset/registry/streamline users to ${db_password} cat < mysql-setup.sql ALTER USER 'root'@'localhost' IDENTIFIED BY 'Secur1ty!'; uninstall plugin validate_password; CREATE DATABASE registry DEFAULT CHARACTER SET utf8; CREATE DATABASE streamline DEFAULT CHARACTER SET utf8; CREATE USER 'registry'@'%' IDENTIFIED BY '${db_password}'; CREATE USER 'streamline'@'%' IDENTIFIED BY '${db_password}'; GRANT ALL PRIVILEGES ON registry.* TO 'registry'@'%' WITH GRANT OPTION ; GRANT ALL PRIVILEGES ON streamline.* TO 'streamline'@'%' WITH GRANT OPTION ; commit; EOF #execute sql file mysql -h localhost -u root -p"$oldpass" --connect-expired-password < mysql-setup.sql #change Mysql password to StrongPassword mysqladmin -u root -p'Secur1ty!' password StrongPassword #test password and confirm dbs created mysql -u root -pStrongPassword -e 'show databases;' 4. On the same node, install Mysql connector jar and then HDF mpack. Then restart Ambari so it recognizes HDF stack: sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar sudo ambari-server install-mpack --mpack=${hdf_ambari_mpack_url} --verbose sudo ambari-server restart At this point, if you wanted you could use Ambari install wizard to install HDF you can do that as well. Just open http://<Ambari host IP>:8080 and login and follow the steps in the doc. Otherwise, to proceed with deploying via blueprints follow the remaining steps. 4. On the same node, provide minimum configurations required for install by creating configuration-custom.json. You can add to this to customize any component's property that is exposed by Ambari cd /tmp/ambari-bootstrap/deploy/ tee configuration-custom.json > /dev/null << EOF { "configurations": { "ams-grafana-env": { "metrics_grafana_password": "${db_password}" }, "streamline-common": { "jar.storage.type": "local", "streamline.storage.type": "mysql", "streamline.storage.connector.connectURI": "jdbc:mysql://$(hostname -f):3306/streamline", "registry.url" : "http://localhost:7788/api/v1", "streamline.dashboard.url" : "http://localhost:9089", "streamline.storage.connector.password": "${db_password}" }, "registry-common": { "jar.storage.type": "local", "registry.storage.connector.connectURI": "jdbc:mysql://$(hostname -f):3306/registry", "registry.storage.type": "mysql", "registry.storage.connector.password": "${db_password}" }, "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "${nifi_password}" }, "nifi-ambari-config": { "nifi.security.encrypt.configuration.password": "${nifi_password}" } } } EOF 5. Then run below as root to generate a recommended blueprint and deploy the cluster install. Make sure to set host_count to the total number of hosts in your cluster (including Ambari server) sudo su cd /tmp/ambari-bootstrap/deploy/ export host_count=<Number of total nodes> export ambari_stack_name=HDF export ambari_stack_version=3.1 export ambari_services="ZOOKEEPER STREAMLINE NIFI KAFKA STORM REGISTRY NIFI_REGISTRY AMBARI_METRICS" ./deploy-recommended-cluster.bash You can now login into Ambari at http://<Ambari host IP>:8080 and sit back and watch your HDF cluster get installed! Notes: a) This will only install Nifi on a single node of the cluster by default b) Nifi Certificate Authority (CA) component will be installed by default. This means that if you wanted to, you could enable SSL to be enabled for Nifi out of the box by including a "nifi-ambari-ssl-config" section in the above configuration-custom.json: "nifi-ambari-ssl-config": { "nifi.toolkit.tls.token": "hadoop", "nifi.node.ssl.isenabled": "true", "nifi.security.needClientAuth": "true", "nifi.toolkit.dn.suffix": ", OU=HORTONWORKS", "nifi.initial.admin.identity": "CN=nifiadmin, OU=HORTONWORKS", "content":"<property name='Node Identity 1'>CN=node-1.fqdn, OU=HORTONWORKS</property><property name='Node Identity 2'>CN=node-2.fqdn, OU=HORTONWORKS</property><property name='Node Identity 3'>node-3.fqdn, OU=HORTONWORKS</property>" }, Make sure to replace node-x.fqdn with the FQDN of each node running Nifi c) As part of the install, you can also have an existing Nifi flow deployed by Ambari. First, read in a flow.xml file from existing Nifi system (you can find this in flow.xml.gz). For example, run below to read the flow for the Twitter demo into an env var twitter_flow=$(curl -L https://gist.githubusercontent.com/abajwa-hw/3a3e2b2d9fb239043a38d204c94e609f/raw) Then include a "nifi-ambari-ssl-config" section in the above configuration-custom.json when you run the tee command - to have ambari-bootstrap include the whole flow xml into the generated blueprint: "nifi-flow-env" : { "properties_attributes" : { }, "properties" : { "content" : "${twitter_flow}" } } d) In case you would like to review the generated blueprint before it gets deployed, just set the below variable as well: export deploy=false .... The blueprint will be created under /tmp/ambari-bootstrap*/deploy/tempdir*/blueprint.json Sample blueprint Sample generated blueprint for 4 node cluster is provided for reference here: { "Blueprints": { "stack_name": "HDF", "stack_version": "3.1" }, "host_groups": [ { "name": "host-group-3", "components": [ { "name": "NIFI_MASTER" }, { "name": "DRPC_SERVER" }, { "name": "METRICS_GRAFANA" }, { "name": "KAFKA_BROKER" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "STREAMLINE_SERVER" }, { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "NIMBUS" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "NIFI_REGISTRY_MASTER" }, { "name": "REGISTRY_SERVER" }, { "name": "STORM_UI_SERVER" } ] }, { "name": "host-group-2", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "ZOOKEEPER_SERVER" } ] }, { "name": "host-group-1", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "NIFI_CA" } ] }, { "name": "host-group-4", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "METRICS_COLLECTOR" }, { "name": "ZOOKEEPER_SERVER" } ] } ], "configurations": [ { "nifi-ambari-config": { "nifi.security.encrypt.configuration.password": "StrongPassword" } }, { "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "StrongPassword" } }, { "ams-hbase-env": { "hbase_master_heapsize": "512", "hbase_regionserver_heapsize": "768", "hbase_master_xmn_size": "192" } }, { "nifi-logsearch-conf": {} }, { "storm-site": { "topology.metrics.consumer.register": "[{\"class\": \"org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink\", \"parallelism.hint\": 1, \"whitelist\": [\"kafkaOffset\\..+/\", \"__complete-latency\", \"__process-latency\", \"__execute-latency\", \"__receive\\.population$\", \"__sendqueue\\.population$\", \"__execute-count\", \"__emit-count\", \"__ack-count\", \"__fail-count\", \"memory/heap\\.usedBytes$\", \"memory/nonHeap\\.usedBytes$\", \"GC/.+\\.count$\", \"GC/.+\\.timeMs$\"]}]", "metrics.reporter.register": "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter", "storm.cluster.metrics.consumer.register": "[{\"class\": \"org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter\"}]" } }, { "registry-common": { "registry.storage.connector.connectURI": "jdbc:mysql://ip-172-31-21-233.us-west-1.compute.internal:3306/registry", "registry.storage.type": "mysql", "jar.storage.type": "local", "registry.storage.connector.password": "StrongPassword" } }, { "registry-logsearch-conf": {} }, { "streamline-common": { "streamline.storage.type": "mysql", "jar.storage.type": "local", "streamline.storage.connector.connectURI": "jdbc:mysql://ip-172-31-21-233.us-west-1.compute.internal:3306/streamline", "streamline.dashboard.url": "http://localhost:9089", "registry.url": "http://localhost:7788/api/v1", "streamline.storage.connector.password": "StrongPassword" } }, { "ams-hbase-site": { "hbase.regionserver.global.memstore.upperLimit": "0.35", "hbase.regionserver.global.memstore.lowerLimit": "0.3", "hbase.tmp.dir": "/var/lib/ambari-metrics-collector/hbase-tmp", "hbase.hregion.memstore.flush.size": "134217728", "hfile.block.cache.size": "0.3", "hbase.rootdir": "file:///var/lib/ambari-metrics-collector/hbase", "hbase.cluster.distributed": "false", "phoenix.coprocessor.maxMetaDataCacheSize": "20480000", "hbase.zookeeper.property.clientPort": "61181" } }, { "ams-env": { "metrics_collector_heapsize": "512" } }, { "kafka-log4j": {} }, { "ams-site": { "timeline.metrics.service.webapp.address": "localhost:6188", "timeline.metrics.cluster.aggregate.splitpoints": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetFetch.98percentile", "timeline.metrics.host.aggregate.splitpoints": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetFetch.98percentile", "timeline.metrics.host.aggregator.ttl": "86400", "timeline.metrics.service.handler.thread.count": "20", "timeline.metrics.service.watcher.disabled": "false" } }, { "kafka-broker": { "kafka.metrics.reporters": "org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter" } }, { "ams-grafana-env": { "metrics_grafana_password": "StrongPassword" } }, { "streamline-logsearch-conf": {} } ] } Sample cluster.json for 4 node cluster: { "blueprint": "recommended", "default_password": "hadoop", "host_groups": [ { "hosts": [ { "fqdn": "ip-172-xx-xx-x3.us-west-1.compute.internal" } ], "name": "host-group-3" }, { "hosts": [ { "fqdn": "ip-172-xx-xx-x2.us-west-1.compute.internal" } ], "name": "host-group-2" }, { "hosts": [ { "fqdn": "ip-172-xx-xx-x4.us-west-1.compute.internal" } ], "name": "host-group-4" }, { "hosts": [ { "fqdn": "ip-172-xx-xx-x1.us-west-1.compute.internal" } ], "name": "host-group-1" } ] } What next? Now that your cluster is up, you can explore what Nifi's Ambari integration means: https://community.hortonworks.com/articles/57980/hdf-20-apache-nifi-integration-with-apache-ambarir.html Next, you can enable SSL for Nifi: https://community.hortonworks.com/articles/58009/hdf-20-enable-ssl-for-apache-nifi-from-ambari.html ... View more

Overview Partner demo kit is built and maintained by the Hortonworks Partner Solutions team. The purpose of the demo kit is to enable the partners to: Quickly bring up a HDP environments with pre-built demos Leverage available demos to understand the capabilities of the platform Use the demos as part of business conversation to demonstrate the art of possible The remainder of this article provides a short description of the 3 demos packaged within the demo kit and step by step instruction on: How to launch the demo kit on AWS or on private cloud How to execute the demos provided with the demo kit Other Versions The Security/Governance Demo kit for HDP 2.6 can be found here The previous version of demo kit (for HDP 2.5) can be found here Pre-requisites When using AWS, you must already have created your Amazon Web Services account. Sample steps for doing this can be found here. If you have an AWS promo code, you can apply it to your account using the steps here. For running the sentiment demo, you must have created a Twitter application using your Twitter account and generated consumer keys/secrets. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Notes Note that the partner demo kit is not a formally supported offering. In case of questions, see ‘Questions?” section at the end of this article. Slides Slides for demo kit are available here Packaged Demos The demo kit comes with 3 demos: 1. IOT demo Purpose: IOT demo showcases how a logistic company uses the Hortonworks Connected Data Platform to monitor its fleet in real time to mitigate driving infractions Use case setup: Sensor devices from trucks capture events of the trucks and actions of the drivers. Some of these driver events are dangerous "events” such as: Lane Departure, Unsafe following distance, Unsafe tail distance The Business Requirement is to stream these events in, filter on violations and do real-time alerting when “lots” of erratic behavior is detected for a given driver over a short period of time. Over time, users would like to do advanced analytics on the full archive of historical events generated by the trucks to: Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc) Build an AI model to make predictions when violations will occur Technologies used: Apache Nifi, Kafka, Storm, Streaming Analytics Manager, Schema Registry, HBase, Spark, Zeppelin More details available here and here 2. Sentiment demo Purpose: Sentiment demo showcases how a retail company can use the Hortonworks Connected Data Platform to visualize and analyze social media data related to their products Use case setup: The Business Requirement is to capture, process and analyze flow of tweets to understand the social sentiments for their products Technologies used: Apache Nifi, Solr, HDFS More details available here and here 3. Advanced analytics demo Purpose: Advanced analytics demo showcases how an insurance company can use the Hortonworks Connected Data Platform to visualize and make predictions on earthquake data using Apache Spark’s machine learning libraries Use case setup: The Business Requirement is to be able to perform advanced analytics on world wide earthquake data to predict where large earthquakes will happen so the business can accordingly modify insurance premiums Technologies used: Apache Spark, Zeppelin More details here Option #1: Installing the Demo Kit on your own setup You can install Demo Kit on other public or private clouds using the provided automated script. With this option you would launch a CentOS/RHEL 7 VM of the right size on any cloud of your choice (as long as it has access to public internet), and use provided script to install single node HDP and install the demo. For more details see README here. Setup ETA is 1 hour Option #2: Launching the Demo Kit AMI on AWS You can use this option to launch a prebuilt image of single node HDP (including the demo) on AWS cloud. Setup ETA is 15min Steps to launch the AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep the default of 500 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For long running instances (i.e. anything beyond an hour), a more restrictive security group policy is strongly encouraged (for example: only allow traffic from your company’s IP range). Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster) . If it is blank, wait 1-2 minutes for this to be populated 12. After 5-10 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as admin user using StrongPassword as password 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. (Optional) You can also monitor the startup using the log as below: Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> Tail the startup log: tail -f /var/log/hdp_startup.log Once you see “cluster is ready!” you can proceed 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. If any services fail to start, use the Actions > Start All button to start 15. At this point you can follow the demo instructions. Troubleshooting If any service does not come up for some reason, you can use Ambari to retry by clicking: ‘Actions’ > ‘Start all’. In case of multiple failures when starting services, use the EC2 dashboard to double check that the correct instance type was used. Insufficient resources can cause services to not start up successfully It is not required to connect via SSH to your instance. But you can do this using the key pair you created/selected earlier by following the standard instructions on AWS website. Make sure the user you login as is centos A log file of the automated startup of HDP services is available under: /var/log/hdp_startup.log Stopping/Terminating demo kit Once you are done with demo kit, we recommend bringing it down to avoid incurring any unnecessary charges. To do this, follow below: First, stop the cluster services using Ambari by clicking: ‘Actions’ > ‘Stop all’. Then pick from one of the two options: a) Terminate the instance: If you do not want to incur any further charges from AWS, terminate the VM instance from the same ‘EC2 dashboard’ that displayed the instance details. Note that this will destroy the VM, so the next time you wish to use demo kit, you will need to follow the same steps outlined in above section ‘Launching the Demo Kit’ b) Stop the instance: if you want to bring down your VM instance but keep it around so you can start it back up in the future, stop the VM instance from the EC2 dashboard. Note that this option will preserve any customizations you make to the VM but you will incur AWS charges by choosing for this option. More details on stop vs terminate operations can be found on AWS website here and here Demo Execution Steps IOT Demo Video recording of the IOT demo Recording of demo provided here (high level) and here (deeper level) PPT and PDF versions of the slides also available IOT Demo setup instructions Sequence to walk through the IOT trucking demo: Events simulator Schema Registry UI NiFi flow SAM Application view Storm Monitoring view Superset Dashboard Superset Slice creation Zeppelin notebook Detailed steps for IOT trucking demo walk through (Optional): Check that events are being simulated. This step is optional because we will also check this from NiFi UI Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> sudo su - To check events being simulated you can either verify the simulator process is running or monitor the simulator log: ps -ef | grep stream-simulator tail -f /tmp/whoville/data_simulator/simulator.log If simulator is not running, you can invoke it by running below from SSH sessioncd /tmp/whoville/data_simulator/sudo ./runDataLoader.sh In case you need to kill the simulator use the ps command above to find the process id and then kill it Next, we will open the web UIs of a number of components that are part of the demo using the Ambari Quicklinks. For example, for Schema Registry here is how to access the Quicklink: Open Schema Registry using Quicklink in Ambari and check 4 schemas below are listed Open NiFi using Quicklink in Ambari, check that “IOT trucking demo” process group is started Double click on the “IOT trucking demo” box to see the details of the flow. The counters should show that simulated events are flowing through the NiFi flow. You can refresh the UI to see this: Open Storm Monitoring view (under Ambari views), and check the topology is live Open SAM using Quicklink in Ambari, check the application is deployed Double click on the application to see more details. You should see that the Emitted and Transferred fields are non-zero (assuming the simulator has been been running for a few min) Open Druid Console using Quicklink in Ambari, check the two datasets are present Open Druid Superset using Quicklink in Ambari and login using admin/StrongPassword There should be one entry under Dashboards. Click it to open the prebuilt dashboard. The prebuilt dashboard will open. You can periodically click the refresh button to see new data arriving. Datasets can take 2-6 mins for new events to appear in Druid The first few slices (i.e graphs) provide monitoring related information (e.g. how many violations? Who are the violators? etc). The last 3 slices provide information about the predictions made by the model (i.e. which routes are predicted to have most violations? Which drivers are predicted to have violations) You can also create other slices and add them to the dashboard using the steps here Optionally you can also demonstrate how a data scientist would use archived truck events to build a model to predict violations. Note, to limit amount of resources needed to run the AMI, Spark/Hive has not been installed so you will not be able to actually run the notebook. The previous version of demokit HDP sandbox has these set up so that can be used if you want to actually execute the steps in the notebook. To walk through the trucking events analysis notebook, first open Zeppelin UI using the Quicklink from Ambari: Login as admin/admin Under Notebook section, use search text field to search for “Trucking data analysis” notebook using Zeppelin search: Click Save on the interpreter binding Walk through the notebook to show how data scientist can use SparkSQL to visualize data to help understand what features should be included in the model Finally you can show that once the important features are known, a model can be built to predict violations (in this case, using Logistical Regression) Stopping/Starting the simulator To stop the simulator, use below command to find its process id and then use kill command to kill it: ps -ef | grep stream-simulator kill <process_id> To start it back up, run below:cd /tmp/whoville/data_simulator/sudo ./runDataLoader.sh Sentiment Demo Video recording of the Sentiment demo Recording of setup instructions for demo provided here Sentiment Demo setup instructions Open Nifi UI using Quicklinks in Ambari Doubleclick "Twitter Dashboard" to open this process group: Right click "Grab Garden Hose" > Properties and enter your Twitter Consumer key/secret and Access token/secret. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Optionally change the 'Terms to filter on' as desired. Once complete, start the flow. Use Banana UI quicklink from Ambari to open Twitter dashboard An empty dashboard will initially appear. After a minute, you should start seeing charts appear Advanced Analytics Demo Video recording of Advanced Analytics demo Video recording provided here Advanced Analytics Demo setup instructions Open Zeppelin UI via Quicklink Login as admin. Password is same as Ambari password A directory structure containing a number of demo notebooks will appear. Find the earthquake demo notebook by filtering for ‘earthquake’ On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults. Now you can walk through the notebook and show the visualizations and process of building the model. Note, to limit amount of resources needed to run the AMI, Spark/Hive has not been installed so you will not be able to actually run the notebook. The previous version of demokit or HDP sandbox has the notebook set up so that can be used if you want to actually execute the steps in the notebook. This concludes this article on how to launch the demo kit and access the provided demonstrations Questions? In case of questions or issues: 1. Search on our Hortonworks Community Connection forum. For example, to find all Demo Kit related posts access this url 2. If you were not able to find the solution, please post a new question using the tag “partner-demo-kit” here. Please try to be as descriptive as possible when asking questions by providing: Detailed description of problem Steps to reproduce problem Environment details e.g. Instance type used was m4.2xlarge Storage used was 500gb Etc Relevant log file snippets ... View more

Newer version available This article describes how to deploy HDP 2.5 version of demokit. While it can still be used, there is now a newer version of demokit available here that leverages combination of HDP 2.6/HDF 3.0 Overview Partner demo kit is built and maintained by the Hortonworks Partner Solutions team. The purpose of the demo kit is to enable the partners to: Quickly bring up a HDP environments with pre-built demos Leverage available demos to understand the capabilities of the platform Use the demos as part of business conversation to demonstrate the art of possible The remainder of this article provides a short description of the 3 demos packaged within the demo kit and step by step instruction on: How to launch the demo kit on AWS or on private cloud How to execute the demos provided with the demo kit Pre-requisites If using AWS, you must already have created your Amazon Web Services account. Sample steps for doing this can be found here. If you have an AWS promo code, you can apply it to your account using the steps here. For running the sentiment demo, you must have created a Twitter application using your Twitter account and generated consumer keys/secrets. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Notes Note that the partner demo kit is not a formally supported offering. In case of questions, see ‘Questions?” section at the end of this article. Slides Slides for the demo kit are available here Webinar Webinar recording about the demo kit available here Packaged Demos The demo kit comes with 3 demos. The slide for these are available here : 1. IOT demo Purpose: IOT demo showcases how a logistic company uses the Hortonworks Connected Data Platform to monitor its fleet in real time to mitigate driving infractions Use case setup: Sensor devices from trucks capture events of the trucks and actions of the drivers. Some of these driver events are dangerous "events” such as: Lane Departure, Unsafe following distance, Unsafe tail distance The Business Requirement is to stream these events in, filter on violations and do real-time alerting when “lots” of erratic behavior is detected for a given driver over a short period of time. Over time, users would like to do advanced analytics on the full archive of historical events generated by the trucks to: Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc) Build an AI model to make predictions when violations will occur Technologies used: Apache Nifi, Kafka, Storm, HBase, Spark, Zeppelin More details available here and here 2. Sentiment demo Purpose: Sentiment demo showcases how a retail company can use the Hortonworks Connected Data Platform to visualize and analyze social media data related to their products Use case setup: The Business Requirement is to capture, process and analyze flow of tweets to understand the social sentiments for their products Technologies used: Apache Nifi, Solr, Hive, HDFS More details available here and here 3. Advanced analytics demo Purpose: Advanced analytics demo showcases how an insurance company can use the Hortonworks Connected Data Platform to visualize and make predictions on earthquake data using Apache Spark’s machine learning libraries Use case setup: The Business Requirement is to be able to perform advanced analytics on world wide earthquake data to predict where large earthquakes will happen so the business can accordingly modify insurance premiums Technologies used: Apache Spark, Zeppelin More details here Launching the Demo Kit Option 1: Launching the Demo Kit on AWS using AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the Demo Kit AMI from ‘N. California’ region by clicking here. Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep the default of 500 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For production usage, a more restrictive security group policy is strongly encouraged. As an instance only allow traffic from your company’s IP range. Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster) and the ‘Owner’ id (which will be the default password). If the ‘Public IP’ is blank, wait 1-2 minutes for this to be populated 12. After about 20 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as admin user using your ‘Owner’ id as password (you can find your owner id in instance details page as highlighted above) 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. 15. At this point you can follow the demo instructions. Troubleshooting If any service does not come up for some reason, you can use Ambari to retry by clicking: ‘Actions’ > ‘Start all’. In case of multiple failures when starting services, use the EC2 dashboard to double check that the correct instance type was used. Insufficient resources can cause services to not start up successfully It is not required to connect via SSH to your instance. But you can do this using the key pair you created/selected earlier by following the standard instructions on AWS website. Make sure the user you login as is ec2-user A log file of the automated startup of HDP services is available under: /var/log/hdp_startup.log Logs of individual HDP components can be found under /var/log/<component name> or can be accessed using Logsearch UI available at http://<PUBLIC IP>:61888 (login with same credentials as Ambari) Stopping/Terminating demo kit Once you are done with demo kit, we recommend bringing it down to avoid incurring any unnecessary charges. To do this, follow below: First, stop the cluster services using Ambari by clicking: ‘Actions’ > ‘Stop all’. Then pick from one of the two options: a) Terminate the instance: If you do not want to incur any further charges from AWS, terminate the VM instance from the same ‘EC2 dashboard’ that displayed the instance details. Note that this will destroy the VM, so the next time you wish to use demo kit, you will need to follow the same steps outlined in above section ‘Launching the Demo Kit’ b) Stop the instance: if you want to bring down your VM instance but keep it around so you can start it back up in the future, stop the VM instance from the EC2 dashboard. Note that this option will preserve any customizations you make to the VM but you will incur AWS charges by choosing for this option. More details on stop vs terminate operations can be found on AWS website here and here Option 2: Installing the Demo Kit on other setups You can also install Demo Kit on other public or private clouds using the provided automated script. 1. Launch a CentOS/RHEL 6 or 7 instance on any cloud of your choice with at least 4 cores and 32 GB RAM. Make sure the instance has access to internet. Warning: Do NOT run this script on an instance where Ambari or HDP has already been installed (including HDP sandbox) 2. SSH into the instance and run the below commands export host_count=1 export stack=HDPDEMO export ambari_password=BadPass#1 ## change password as needed curl -sSL https://gist.github.com/abajwa-hw/3f2e211d252bba6cad6a6735f78a4a93/raw | sudo -E sh 3. This will install Ambari server and agents. After 5-10 min, you should get a message saying the blueprint was deployed which means cluster install has started. At this point you can login to Ambari UI on port 8080 (using user: admin and whatever password you specified above) and monitor the cluster install/startup Demo Execution Steps IOT Demo Video recording of the IOT demo Recording of demo provided here PPT and PDF versions of the slides also available Recording of setup instructions for demo provided here IOT Demo setup instructions Part 1: We will be using the IOT trucking web application to deploy a Storm topology. Then we will use Nifi to push simulated trucking events into Kafka where they will be pulled by Storm for windowing analysis, before being pushed out to a web app and HBase In Ambari, open 'IotDemo UI' using quicklink: In IotDemo UI, click "Deploy the Storm Topology" After 30-60 seconds, the topology will be deployed. Confirm using the Storm View in Ambari: Click "Truck Monitoring Application" link in 'IotDemo UI' to open the monitoring app showing an empty map. Click 'Nifi Data Flow' in In IotDemo UI to launch Nifi and then double click on 'Iot Trucking demo' processor group. The flow should already be started so no action needed. In Ambari, click "Generate Events" to simulate 50 events (this can be configured) Switch back to "Truck Monitoring Application" in IotDemo UI and after 30s the trucking events will appear on screen Examine the Storm topology using Storm View in Ambari Part 2: Next, you can run through the Trucking data Zeppelin notebook to do advanced analytics on the full archive of historical events generated by the trucks to Determine what factors have an impact on driving violations (e.g. weather, driver fatigue etc) Build an AI model to make predictions when violations will occur Login to Zeppelin interface using the steps provided under the below section: ‘Advanced Analytics Demo setup instructions’ Find and open the ‘Trucking Data Analysis’ notebook by filtering for ‘truck’ On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults. Execute the code cells one by one, by clicking the 'Play' (triangular) button on top right of each cell. Alternatively you can just highlight a cell then press Shift-Enter You can tell that the status of the cell is RUNNING by the label on the top right of the cell. Note that the first invocation of a cell that runs Spark takes 30-60 seconds as the Spark Application Master is launched on YARN. If desired, you can monitor this using YARN’s Resource Manager UI and Spark UI (for detailed steps, see the below section ‘Advanced Analytics Demo setup instructions’) Sentiment Demo Video recording of the Sentiment demo Recording of setup instructions for demo provided here Slides available here Sentiment Demo setup instructions Open Nifi UI using Quicklinks in Ambari Doubleclick "Twitter Dashboard" to open this process group: Right click "Grab Garden Hose" > Properties and enter your Twitter Consumer key/secret and Access token/secret. If you do not have these, you can generate a new set using your Twitter account by following this section of the Hortonworks tutorial. Optionally change the 'Terms to filter on' as desired. Once complete, start the flow. Use Banana UI quicklink from Ambari to open Twitter dashboard An empty dashboard will initially appear. After a minute, you should start seeing charts appear Use Hive view in Ambari run SQL queries on tweet data Advanced Analytics Demo Video recording of Advanced Analytics demo Video recording provided here Slides provided here Advanced Analytics Demo setup instructions Open Zeppelin UI via Quicklink Login as admin. Password is same as Ambari password A directory structure containing a number of demo notebooks will appear. Find the earthquake demo notebook by filtering for ‘earthquake’ On first launch of a notebook, you will see that the "Interpreter Binding" settings will be displayed. You will need to click "Save" under the interpreter order to accept the defaults. Execute the code cells one by one, by clicking the 'Play' (triangular) button on top right of each cell. Alternatively you can just highlight a cell then press Shift-Enter You can tell that the status of the cell is RUNNING by the label on the top right of the cell. Note that the first invocation of a cell that runs Spark takes 30-60 seconds as the Spark context is launched. Under the covers it is launching a Spark Application Master on YARN. If desired, you can monitor this using Resource Manager UI which is available through Ambari under Yarn > Quicklink. The Spark UI can also be access from Resource Manager UI by clicking on application ID for Zeppelin app and then click on ‘Application Master’ hyperlink under ‘Tracking Url’ The Spark UI can be used to monitor the running Spark jobs ’ This concludes this article on how to launch the demo kit and access the provided demonstrations Questions? In case of questions or issues: 1. Search on our Hortonworks Community Connection forum. For example, to find all Demo Kit related posts access this url 2. If you were not able to find the solution, please post a new question using the tag “partner-demo-kit” here. Please try to be as descriptive as possible when asking questions by providing: Detailed description of problem Steps to reproduce problem Environment details e.g. Instance type used was m4.2xlarge Storage used was 500gb Etc Relevant log file snippets ... View more