Overview There are many useful articles as well official Cloudbreak documentation covers everything in great depth. This short article walks you through how to deploy the Cloudbreak instance within the existing VPC and Subnet using the AWS Quickstart deployment. Cloudbreak deployment options The cloudbreak deployment options are explained in detail here. If you notice the AWS specific Networking options, the Quickstart by default creates the new VPC and for the custom VPC the production install is recommended. In case you are doing poc and quickly want to try the Quickstart option but wanted to use the existing VPC, you can do that by enhancing the CloudFormation template which is described in next session. CloudFormation template changes When you launch the CloudFormation template for AWS Quickstart by default it selects the existing CloudFormation template https://s3.amazonaws.com/cbd-quickstart/cbd-quickstart-2.7.0.template. Instead of using the default template use the following template. https://github.com/mpandithw/cloudbreak/blob/master/CloudFormation_aws_QuickStart-Template Mainly following changes have been made to the original template. Added the following two parameters VpcId and SubnetId. "VpcId": { "Type": "AWS::EC2::VPC::Id", "Description": "VpcId of your existing Virtual Private Cloud (VPC)" },"SubnetId": { "Type": "AWS::EC2::Subnet::Id", "Description": "SubnetId of your existing Virtual Private Cloud (VPC)" } I am not walking through all the detail steps which are already covered in cloudbreak documentation. Only modification to original process is to select your own CloudFormation template which is described above. You will get the options with drop down list of existing VPC and subnet. Complete the rest of process as explained in the cloudbreak documentation. Benefits You can use the AWS Quickstart deployment of cloudbreak within your existing VPC/Subnet. Document References aws CloudFormation user guide ... View more

This is the continuation of article Part-1 provisioning HDP/HDF cluster on google cloud. Now that we have Google credentials created, we can provision the HDP/HDF cluster. Lets first start with HDP cluster. Login to Cloudbreak UI and click on create cluster which will open the create cluster wizard with both basic and advanced options. On the general configuration page Select the previously created Google credentials, Enter name of the cluster , Select region as shown below, Select either HDP or HDF version. For cluster type select the appropriate cluster blueprint based on your requirements. The available blueprint option in cloudbreak 2.5 tech preview are shown below. Next is configuring the Hardware and Storage piece. Select the Google VM instance type from the dropdown. Enter number of instances for each group. You must select one node for ambari server for one of the host group for which the Group Size should be set to "1". Next is setup the Network group. You can select the existing network or you have option to create new network. On the Security config page provide the cluster admin username and password. Select the new ssh key public key option or the existing ssh public key option. You will use the matching private key to access your nodes via SSH. Finally you will hit create cluster which will redirect you to cloudbreak dashboard. The following left image shows the cluster creation in progress and right image shows the successfully creation of HDP cluster on Google cloud. Once successful deploying the HDP cluster you can login to HDP nodes using your ssh private key with choice of your tool. Following image shows the node login using google cloud browser option. Similarly you can provision the HDF (NiFi: Flow management ) cluster using cloudbreak which is included as part of 2.5 tech preview. Following are some key screenshots for the reference. The Network, Storage and security configuration is similar as we have seen in HDP section earlier. With limitation with my google cloud account subscription I ran into the exception while creating HDF cluster which was rightly shown on cloudbreak. I had to select different region to resolve it. The nifi cluster got created successfully as shown below. Conclusion: Cloudbreak can provide you the easy button to provision and monitor the connected data platform (HDP and HDF) in the cloud vendor of your choice to build the modern data applications. ... View more

Cloudbreak Overview Overview Cloudbreak enables enterprises to provision Hortonworks platforms in Public (AWS + GCP + Azure) and Private (OpenStack) cloud environments. It simplifies the provisioning, management, and monitoring of on-demand HDP and HDF clusters in virtual and cloud environments. Following are primary use cases for Cloudbreak: Dynamically configure and manage clusters on public or private clouds. Seamlessly manage elasticity requirements as cluster workloads change Supports configuration defining network boundaries and configuring security groups. This article focuses on deploying HDP and HDF cluster on Google Cloud. Cloudbreak Benefits You can spin up connected data platform (HDP and HDF clusters) on choice of your cloud vendor using open source Cloudbreak 2.0 which address the following scenarios. Defining the comprehensive Data Strategy irrespective of deployment architecture (cloud or on premise). Addressing the Hybrid (on-premise & cloud) requirements. Supporting the key Multi-cloud approach requirements. Consistent and familiar security and governance across on-premise and cloud environments. Cloudbreak 2 Enhancements Recently Hortonworks announced the general Availability of the Cloudbreak 2.4 release. Following are some of the major enhancements in the Cloudbreak 2.4: New UX / UI: a greatly simplified and streamlined user experience. New CLI: a new CLI that eases automation, an important capability for cloud DevOps. Custom Images: advanced support for “bring your own image”, a critical feature to meet enterprise infrastructure requirements. Kerberos: ability to enable Kerberos security on your clusters, must for any enterprise deployment. You can check the following HCC article for detail overview of Cloudbreak 2.4 https://community.hortonworks.com/articles/174532/overview-of-cloudbreak-240.html Also check the following article for the Cloudbreak 2.5 tech preview details. https://community.hortonworks.com/content/kbentry/182293/whats-new-in-cloudbreak-250-tp.html Prerequisites for Google Cloud Platform. Article assumes that you have already installed and launch the Cloudbreak instance either on your own custom VM image or on Google Cloud Platform. You can follow the Cloudbreak documentation which describes both the options. https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.5.0/content/index.html https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.5.0/content/gcp-launch/index.html In order to launch the Cloudbreak and provision the clusters make sure you have the Google cloud account. You can create one at https://console.cloud.google.com Create new project in GCP (e.g. GCPIntegration project as shown below). In order to launch the clusters on GCP you must have service account that Cloudbreak can use. Assign the admin roles for the Compute Engine and Storage. You can check the required service account admin roles at Admin Roles Make sure you create the P12 key and store it safely. This article assumes that you have successfully meet the prereqs and able to launch the cloudbreak UI as shown left below by visiting https://<IP_Addr or HostName> and Upon successful login you are redirected to the dashboard which looks like the image on right. Create Cloudbreak Credential for GCP. First step before provisioning cluster is to create the Cloudbreak credential for GCP. Cloudbreak uses this GCP credentials to create the required resources on GCP. Following are steps to create GCP credential: In Cloudbreak UI select credentials from Navigation pane and click create credentials. Under cloud provider select Google Cloud Platform. As shown below provide the Google project id, Service Account email id from google project and upload the P12 key that you created the above section. Once you provide all the right details , cloudbreak will create the GCP credential and that should be displayed in the Credential pane. Next article Part 2 covers in detail how to provision the HDP and HDF cluster using the GCP credential. ... View more

Introduction Recently worked with use case which required heavy xml processing. Instead of writing complex custom code end up achieved everything easily with NiFi. I thought this will be useful of someone interested for XML processing in NiFi. The document in general covers the following. Base64 Encoding and Decoding of XML message. Character Set conversion from UTF to Ascii ISO-8859-1 XML validation against the XSD. Split the XML into smaller chunks. Transform XML to JSON. Extract the content and outputs into unique files based on content. This is very generic XML processing flow which can be leveraged across many business use cases which process xml data. Apache NiFi Flow In the sample demo scenario, External system sends the Base64 encoded XML data in file format which is read through GetFile processor. Next Base64EncodeContent processor decoded the Base64 content. Incoming data in UTF-8 format with leading BOM bytes which gets converted to the ISO-8859-1 format using the ConvertCharacterSet processor. XML content is validated against the XML schema using ValidateXML processor. The validated XML fragment splits at the root’s children level into smaller XML chunks. The split xml is converted into JSON object using the XSLT and further written into individual files. Every file is named based on the unique identifier from the flow content. Processor Configurations Base64EncodeContent ConvertCharacterSet ValidateXml: Value :/Users/mpandit/jdeveloper/mywork/ClaimProcess/ClaimProcess/Initiate_App.xsd SplitXml: TransformXMLToJSON: EvalutateJsonPath UpdateAttribute Sample Input and Outputs Input Base64 Encoded XML: PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiID8+DQo8cGVyc29ucyB4bWxuczp4 c2k9Imh0dHA6Ly93d3cudzMub3JnLzIwMDEvWE1MU2NoZW1hLWluc3RhbmNlIiB4c2k6bm9OYW1l c3BhY2VTY2hlbWFMb2NhdGlvbj0iaGVhZGVyLnhzZCI+DQogIDxwZXJzb24+DQogICAgPGZ1bGxf bmFtZT5NUDwvZnVsbF9uYW1lPg0KICAgIDxjaGlsZF9uYW1lPkFCPC9jaGlsZF9uYW1lPg0KICA8 L3BlcnNvbj4NCiAgPHBlcnNvbj4NCiAgICA8ZnVsbF9uYW1lPkdQPC9mdWxsX25hbWU+DQogICAg PGNoaWxkX25hbWU+Q0Q8L2NoaWxkX25hbWU+DQogIDwvcGVyc29uPg0KICA8cGVyc29uPg0KICAg IDxmdWxsX25hbWU+SlA8L2Z1bGxfbmFtZT4NCiAgICA8Y2hpbGRfbmFtZT5FRjwvY2hpbGRfbmFt ZT4NCiAgPC9wZXJzb24+ICANCjwvcGVyc29ucz4= Base64 Decoded XML through NiFi: <?xml version="1.0" encoding="UTF-8" ?> <persons xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="header.xsd"> <person> <full_name>MP</full_name> <child_name>AB</child_name> </person> <person> <full_name>GP</full_name> <child_name>CD</child_name> </person> <person> <full_name>JP</full_name> <child_name>EF</child_name> </person> </persons> Output split XML fragments: Message 1: <?xml version="1.0" encoding="UTF-8"?><person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <full_name>MP</full_name> <child_name>AB</child_name> </person> Message 2: <?xml version="1.0" encoding="UTF-8"?><person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <full_name>GP</full_name> <child_name>CD</child_name> </person> Message 3: <?xml version="1.0" encoding="UTF-8"?><person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <full_name>JP</full_name> <child_name>EF</child_name> </person> JSON output Files: File output 1: { "person" : { "full_name" : "GP", "child_name" : "CD" }} File output 2: { "person" : { "full_name" : "MP", "child_name”: "AB" }} File output 3: { "person" : { "full_name" : "JP", "child_name" : "EF" }} Testing NiFi DataFlow Drop the base 64 encoded XML file which will be processed and split into smaller JSON representation of xml data into individual files. Apache NiFi Benefits In built NiFi processors significantly eliminates the need for custom code to process XML messages. Handles multi byte character sets efficiently expanding range of character set support. The generic XML processing flow templates can accelerate the overall development process. Document References https://nifi.apache.org/docs/nifi-docs/ ... View more

Introduction Hortonworks Dataflow (HDF) powered by Apache NiFi, kafka and Storm, collects, curates, analyzes and delivers real-time data from the IoAT to data stores both on-premises and in the cloud. Apache NiFi automates and manages the flow of information between systems. NiFi data flows are made of series of processors each with specific task. NiFi provides hundreds of general purpose processors. NiFi can pull data from various sources. This document mainly discusses in detail the integration with AWS S3 data source. Amazon S3 is cloud storage for the Internet. To upload your data (photos, videos, documents etc.), you first create a bucket in one of the AWS regions. You can then upload any number of objects to the bucket. S3 buckets and objects are resources, and Amazon S3 provides APIs for you to manage them. With many existing AWS customers there is need to integrate with S3 to process data across multiple applications. NiFi provides many processors to manage and process S3 objects integrating with S3 buckets. This document outlines the detail setup and configuration to integrate S3 with Apache NiFi. Business Use cases Many customers are utilizing the Amazon S3 storage service to build applications.The application types can be backup & archiving, content storage, big data analytics, cloud native application data or DR. S3 can be utilized as persistent or temporary storage. Many applications need to process the data as lands into a S3 bucket, retrieve the content and log the metadata regarding the S3 object. AWS supports following destinations where it can publish S3 related events. Amazon SNS topic Amazon SQS queue AWS lambda This document describes putting and extracting data object from amazon S3 using Apache NiFi leveraging the Amazon SQS notifications. Solution Architecture with Apache NiFi The main purpose of the document is to showcase the ease of integration with S3 using Apache NiFi. In the sample demo scenario: The cloud NiFi instance creates the data object to S3 bucket using PutS3Object processor. As the new object is created at S3, it sends out notification in JSON object form to amazon SQS queue. GetSQS processor subscribes to the SQS event and retrieves the newly created S3 object metadata. The FetchS3Object extracts the content of newly created object and sends it downstream systems for further processing. AWS Configurations This section describes the setup and configuration on AWS side (SQS & S3 bucket) based on the scenario described in the previous section. Make sure to login to the AWS dashboard and select appropriate product section to configure the S3 buckets and SQS queues. SQS One of the ways to monitor S3 bucket is to use SQS notifications. First create the SQS Queue as shown below e.g. NiFiEvent. Configure the security and appropriate permissions to the SQS queue so that SQS queue can be utilized for the S3 bucket events. S3 Bucket Once the SQS configuration is done, create the S3 bucket (e.g. mphdf). Adding a folder named "orderEvent" to the S3 bucket. Go to the properties section and make sure to configure Permissions, Event notification and policy to the S3 bucket. For permissions, add the appropriate account to include list, upload, delete, view and Edit permissions to the bucket. Creating an AWS bucket policy, so that specific accounts have permissions to manage the S3 bucket objects. The policy output will be JSON object which you can update in the bucket configuration. Next configure the event notification to publish all the relevant events to SQS queues which we created earlier. Make sure to select the Type as SQS. The AWS side configuration is now complete. Next, build the NiFi dataflow using the NiFi processors for S3 buckets and SQS. NiFi DataFlow Configuration To demonstrate the S3 integration I modified the existing NiFi data flow. There are two NiFi dataflows, one to publish the object to S3 and second one is to extract the object content through SQS notification. Flow 1 : Accept the ERP Events --> transform to JSON --> PutS3Object Flow 2 : GetSQS notification -->Get Object metadata --> FetchS3Object The complete flow is shown below. PutS3Object Processor This processor creates the object in S3 bucket. The configuration is shown below. Make sure to use the correct bucket name, owner (aws account) , access key and secret key. To get the Access and secret keys go the AWS IAM console -> users ->security credentials->create access keys. You can download or select Show User Security Credentials options to get Access key and Secret access key. Make sure to select the correct Region. GetSQS Processor IT fetches messages from an AWS SQS queue. The Queue URL you can get from AWS SQS details section. FetchS3Object Processor Retrieves the contents of an S3 Object and writes it to the content of a FlowFile. The "object key" is nothing but the key attribute value from incoming SQS notification JSON message. The "objectname" property is populated by JsonPath expression $.Records[*].s3.object.key Testing NiFi DataFlow For testing purpose, the existing dataflow is used which: pulls the ERP events from JMS queue. The event xml is mapped to JSON object and the object is pushed to AWS S3 bucket. The successful creation of S3 object triggers a notification to SQS queue. The second NiFi dataflow reads the SQS event through GetSQS processor. It parses the JSON metadata information and extracts the object name from SQS message. Next it uses the fetchS3Object NiFi processor to extract the S3 object content and then writes it to the local file system. Following are some data provenance screenshots from NiFi which proves the successful creation of S3 object and then processing the SQS notification to extract and process the S3 object through NiFi in near real time fashion. PutS3Object data provenance S3 Bucket browser view (shows the newly created object). GetSQS data provenance Showed the SQS event getting triggered for newly created S3 object. Sample SQS Notification message The highlighted value shows the object name which will be extracted and used to retrieve the object from S3 bucket. {"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-08-03T14:25:13.147Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"AWS:AIDAJL3JQI6HZAG3MB6JM"},"requestParameters":{"sourceIPAddress":"71.127.248.137"},"responseElements":{"x-amz-request-id":"9A673206F1BFDE85","x-amz-id-2":"UmcqEKQJyXfH+UlgDTWIMfvQDOhuOWREe/lwUSJdMx9CbgCu7wzPWJL+wCeRzL6dgqsnYTopWrM="},"s3":{"s3SchemaVersion":"1.0","configurationId":"NiFiEvents","bucket":{"name":"mphdf","ownerIdentity":{"principalId":"A1O76DMOXCPR44"},"arn":"arn:aws:s3:::mphdf"},"object":{"key":"orderEvent/651155932749796","size":804,"eTag":"53ca61e19b3223763f1b36ed0e9383fa","sequencer":"0057A1FEC9137FBF3A"}}}]} FetchS3Bucket data provenance The content from the newly created S3 object extracted using the FetchS3Bucket processor. Apache NiFi Benefits NiFi is data source and destination-agnostic. it can move data from any data platform to any data platform. The S3 example demonstrates it can integrate well with non Hadoop echo system as well. The NiFi project provides connection processors for many standard data sources like S3. With standard interfaces new processors can be developed with minimal effort. You can create NiFi dataflow templates to accelerate development. Apache NiFi is ideal for data sources sitting out on the edge in the cloud or on-prem. Document References NiFi System Admin Guide: http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF1/HDF-1-trunk/bk_AdminGuide/content/ch_administration_guide.html https://aws.amazon.com/s3/?nc2=h_l3_sc https://aws.amazon.com/sqs/?nc2=h_m1 https://nifi.apache.org/docs/nifi-docs/ ... View more

HDF Overview Overview Hortonworks DataFlow (HDF) powered by Apache NiFi, Kafka and Storm, collects, curates, analyzes and delivers real-time data from the IoAT to data stores both on-premises and in the cloud. This is the quick installation guide to install Apache NiFi on AWS EC2 instance. Please refer this document as supplement guide to official Hortonworks HDF documentation. Prerequisites Before you install Apache NiFi on AWS, make sure You have AWS account. (https://aws.amazon.com/) Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair) Installation Steps. The screenshots in this section detail the setup and configuration of Apache NiFi on EC2 instance. Refer the NiFi Admin Guide for the System requirements. This document covers installation on a Redhat linux (64 bit) EC2 instance. Login to AWS and launch the EC2 instance and OS of your choice. (Please make sure the selected OS is supported by NiFi). The current exercise uses the Red Hat Enterprise Linux 7.2 image (HDF EC2 Instance). Make sure you keep the security private key safe. Under Network and Security configuration, open the Http ports (e.g. 8081 & 8082 shown below) to access the NiFi web interface and for the site to site protocol to exchange data between multiple NiFi instances. Download the HDF from HDF Download Page. Either you can download it directly on your EC2 instance or you can upload the zip file to the EC2 instance from local using scp. e.g. scp -i HDF.pem HDF-1.2.0.1-1.zip ec2-user@<public-dns-hostname>:/home/ec2-user where HDF.pem is private key. Make sure you install the latest java and unzip on EC2 sudo yum install unzip sudo yum install java Decompress/Unzip zip into desired installation directory. Make desired edits in nifi.properties file under <install_dir>/nifi/conf. e.g. update the site-to-site properties to include the following nifi.remote.input.socket.host=<public_dns_hostname> nifi.remote.input.socket.port=8082 nifi.remote.input.secure=false From the <install_dir>/nifi/bin directory execute the following commands by ./nifi.sh <command> start: starts NiFi in the background stop: stops NiFi that is running in the background status: provides the current status of NiFi run: runs NiFi in the foreground and waits for a Ctrl-C to initiate shutdown of NiFi install: installs NiFi as a service that can then be controlled via service nifi start service nifi stop service nifi status The following screenshots displays the NiFi running on EC2 instance with the sample dataflow. Benefits Running a NiFi instance in AWS provides an easy to use, flexible and cost effective dataflow management solution in cloud. NiFi is a reliable, secure and scalable solution which gets additional benefits of AWS’ mature infrastructure solution. Using the NiFi site-to-site protocol eliminates the need to run software in the DMZ when exchanging data between on-prem and cloud. Document References NiFi System Admin Guide: http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF1/HDF-1-trunk/bk_AdminGuide/content/ch_administration_guide.html ... View more

Hortonworks Cloud Overview Overview Hortonworks Cloud (Hortonworks Connected Data cloud) is system for launching and managing a data lake on AWS. You can easily launch HDP clusters for analyzing and processing data. With Apache Hadoop. Use this document as supplement guide to existing official guide by Hortonworks. The doc will guide you in running HDP platform in your AWS environment. Prerequisites Before you launch hortonworks cloud controller make sure You have AWS account. (https://aws.amazon.com/) Amazon key pair to access EC2 instance to run HDP platform. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#having-ec2-create-your-key-pair) Hortonworks cloud supported in following regions for now. Us-east-1, us-west-2, eu-central-1, eu-west-1 and ap-northeast-1. Architecture Overview Cloud Controller is the primary component of hortonworks cloud. Cloud controller manages one or more HDP clusters on AWS. It runs on EC2 instance and used for launching and managing cluster. The master and worker nodes runs on multiple EC2 instances. Different cluster type templates are provided to accelerate the HDP platform setup on AWS. Currently two HDP versions are available. HDP 2.4 HDP 2.5 (Technical Preview). During cluster creation you can select the HDP version and cluster types. Cloud Controller The screenshots in this section covers the detail setup and configuration after launching the cloud controller to create the AWS resources. You can launch the cloud controller at http://hortonworks.github.io/hdp-aws/launch/ Setup & Configuration Details. hc1-at-115206-am.png hc2-at-120519-pm.png screen-shot-2016-07-26-at-120243-pm.png screen-shot-2016-07-26-at-120546-pm.png screen-shot-2016-07-26-at-120743-pm.png Once the stack creation is complete you can access the cloud controller to create HDP cluster. Accessing Cloud UI. Using the CloudURL you can access the cloud controller instance. Check the screenshot below to get your CloudURL. screen-shot-2016-07-26-at-10434-pm.png screen-shot-2016-07-26-at-10512-pm.png screen-shot-2016-07-26-at-10544-pm.png You have provided with cluster templates to accelerate the HDP cluster configuration. You can select the HDP version and cluster type. screen-shot-2016-07-26-at-10654-pm.png screen-shot-2016-07-26-at-10821-pm.png screen-shot-2016-07-26-at-10907-pm.png It normally takes less than 30 minutes to setup 3 node HDP cluster. screen-shot-2016-07-26-at-10956-pm.png Validate all the steps for any errors and you can launch the Ambari web UI to verify the cluster setup. screen-shot-2016-07-26-at-30332-pm.png screen-shot-2016-07-26-at-13405-pm.png On AWS dashboard it will provide the details of all the running EC2 instances. screen-shot-2016-07-26-at-13527-pm.png Post cluster setup Actions. The cluster action provides you the option to Resize the cluster Clone the cluster. screen-shot-2016-07-26-at-31618-pm.pngscreen-shot-2016-07-26-at-31339-pm.png Benefits Hortonworks cloud provides greater agility through faster HDP cluster deployment in AWS. It provides elastic scalability on demand You can launch the enterprise ready HDP platform in less than hour. Document References http://hortonworks.github.io/hdp-aws/ If you run into AWS related resource limitation, please validate the following doc. (elastic IP address limit etc.) http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html you can submit request to increasing the limit by submit a request ... View more