HDF Overview Overview Hortonworks DataFlow (HDF) powered by Apache NiFi, Kafka and Storm, collects, curates, analyzes and delivers real-time data from the IoAT to data stores both on-premises and in the cloud. This is the quick installation guide to install Apache NiFi on AWS EC2 instance. Please refer this document as supplement guide to official Hortonworks HDF documentation. Prerequisites Before you install Apache NiFi on AWS, make sure You have AWS account. (https://aws.amazon.com/) Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair) Installation Steps. The screenshots in this section detail the setup and configuration of Apache NiFi on EC2 instance. Refer the NiFi Admin Guide for the System requirements. This document covers installation on a Redhat linux (64 bit) EC2 instance. Login to AWS and launch the EC2 instance and OS of your choice. (Please make sure the selected OS is supported by NiFi). The current exercise uses the Red Hat Enterprise Linux 7.2 image (HDF EC2 Instance). Make sure you keep the security private key safe. Under Network and Security configuration, open the Http ports (e.g. 8081 & 8082 shown below) to access the NiFi web interface and for the site to site protocol to exchange data between multiple NiFi instances. Download the HDF from HDF Download Page. Either you can download it directly on your EC2 instance or you can upload the zip file to the EC2 instance from local using scp. e.g. scp -i HDF.pem HDF-1.2.0.1-1.zip ec2-user@<public-dns-hostname>:/home/ec2-user where HDF.pem is private key. Make sure you install the latest java and unzip on EC2 sudo yum install unzip sudo yum install java Decompress/Unzip zip into desired installation directory. Make desired edits in nifi.properties file under <install_dir>/nifi/conf. e.g. update the site-to-site properties to include the following nifi.remote.input.socket.host=<public_dns_hostname> nifi.remote.input.socket.port=8082 nifi.remote.input.secure=false From the <install_dir>/nifi/bin directory execute the following commands by ./nifi.sh <command> start: starts NiFi in the background stop: stops NiFi that is running in the background status: provides the current status of NiFi run: runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi install: installs NiFi as a service that can then be controlled via service nifi start service nifi stop service nifi status The following screenshots displays the NiFi running on EC2 instance with the sample dataflow. Benefits Running a NiFi instance in AWS provides an easy to use, flexible and cost effective dataflow management solution in cloud. NiFi is a reliable, secure and scalable solution which gets additional benefits of AWS’ mature infrastructure solution. Using the NiFi site-to-site protocol eliminates the need to run software in the DMZ when exchanging data between on-prem and cloud. Document References NiFi System Admin Guide: http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF1/HDF-1-trunk/bk_AdminGuide/content/ch_administration_guide.html ... View more

Hortonworks Cloud Overview Overview Hortonworks Cloud (Hortonworks Connected Data cloud) is system for launching and managing a data lake on AWS. You can easily launch HDP clusters for analyzing and processing data. With Apache Hadoop. Use this document as supplement guide to existing official guide by Hortonworks. The doc will guide you in running HDP platform in your AWS environment. Prerequisites Before you launch hortonworks cloud controller make sure You have AWS account. (https://aws.amazon.com/) Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair) Hortonworks cloud supported in following regions for now. Us-east-1, us-west-2, eu-central-1, eu-west-1 and ap-northeast-1. Architecture Overview Cloud Controller is the primary component of hortonworks cloud. Cloud controller manages one or more HDP clusters on AWS. It runs on EC2 instance and used for launching and managing cluster. The master and worker nodes runs on multiple EC2 instances. Different cluster type templates are provided to accelerate the HDP platform setup on AWS. Currently two HDP versions are available. HDP 2.4 HDP 2.5 (Technical Preview). During cluster creation you can select the HDP version and cluster types. Cloud Controller The screenshots in this section covers the detail setup and configuration after launching the cloud controller to create the AWS resources. You can launch the cloud controller at http://hortonworks.github.io/hdp-aws/launch/ Setup & Configuration Details. hc1-at-115206-am.png hc2-at-120519-pm.png screen-shot-2016-07-26-at-120243-pm.png screen-shot-2016-07-26-at-120546-pm.png screen-shot-2016-07-26-at-120743-pm.png Once the stack creation is complete you can access the cloud controller to create HDP cluster. Accessing Cloud UI. Using the CloudURL you can access the cloud controller instance. Check the screenshot below to get your CloudURL. screen-shot-2016-07-26-at-10434-pm.png screen-shot-2016-07-26-at-10512-pm.png screen-shot-2016-07-26-at-10544-pm.png You have provided with cluster templates to accelerate the HDP cluster configuration. You can select the HDP version and cluster type. screen-shot-2016-07-26-at-10654-pm.png screen-shot-2016-07-26-at-10821-pm.png screen-shot-2016-07-26-at-10907-pm.png It normally takes less than 30 minutes to setup 3 node HDP cluster. screen-shot-2016-07-26-at-10956-pm.png Validate all the steps for any errors and you can launch the Ambari web UI to verify the cluster setup. screen-shot-2016-07-26-at-30332-pm.png screen-shot-2016-07-26-at-13405-pm.png On AWS dashboard it will provide the details of all the running EC2 instances. screen-shot-2016-07-26-at-13527-pm.png Post cluster setup Actions. The cluster action provides you the option to Resize the cluster Clone the cluster. screen-shot-2016-07-26-at-31618-pm.pngscreen-shot-2016-07-26-at-31339-pm.png Benefits Hortonworks cloud provides greater agility through faster HDP cluster deployment in AWS. It provides elastic scalability on demand You can launch the enterprise ready HDP platform in less than hour. Document References http://hortonworks.github.io/hdp-aws/ If you run into AWS related resource limitation, please validate the following doc. (elastic IP address limit etc.) http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html you can submit request to increasing the limit by submit a request ... View more