Introduction - Here is a small demo how to flex MiNiFi+NiFi on a Raspberry Pi to detect Motion and send alerts to your phone via SMS, also on your SMS reply it will trigger sound ALARM [Basically to shoo away an intruder]. You can do it from where ever you have cell phone reception. - Here you can view the screen recording session that Demonstrates how it works! Prerequisite Raspberry Pi 3, a PIR motion sensor and a speaker connected to it. Details on how to connect PIR Motion Sensor to Raspberry Pi can be found in url under references. Assuming you already have latest version of HDF/NiFi and Minifi downloaded on your Mac and Pi. Else Get Latest version of MiNiFi : # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-0.1.0-bin.tar.gz Get Latest version of MiNiFi ToolKit: # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-toolkit-0.1.0-bin.tar.gz Get Latest version of NiFi: # wget http://apache.claz.org/nifi/1.2.0/nifi-1.2.0-bin.tar.gz Untar the files and start NiFi on your local machine and MiniFi on your Raspberry Pi Steps: Flow on MiNiFi Download the flow Pi-MiNiFi-FLow.xml and convert it to YAML format which MiNiFi uses (before you deploy make sure you have your local NiFi URL for RPG rather than what I have in there) # /root/minifi-toolkit-0.1.0/bin/config.sh transform Pi-MiNiFi-FLow.xml minifi-0.1.0/conf/config.yml Flow Looks like below: Flow Explained: a) Poll for Sensor output and Sent it to NiFi GenerateFlowFile processor triggers every 5seconds to execute a python script pirtest.py as below using an ExecuteStreamCommand proceesor, result is sent to NiFi running on my local machine via Remote Process Group. pirtest.py script looks like below: import RPi.GPIO as GPIO import time import os GPIO.setwarnings(False) GPIO.setmode(GPIO.BOARD) GPIO.setup(11, GPIO.IN) #Read output from PIR motion sensor i=GPIO.input(11) if i==1: #When output from motion sensor is HIGH print "Intruder detected",i b) Play Panic Alarm on SMS trigger from NiFi listenHTTP processor hosts and listens for any incoming flowfile, when arrived next processor ExecuteStreamCommand executes a python script alarm.py as given below which trigger a panic alarm sound to be played. I used mpg123 to play the alarm sound, you can install it on your raspberry pi using below command: # sudo apt-get install mpg123 alarm.py script looks like below: import os os.system('mpg123 /root/alarm.mp3') Flow on NiFi Download the flow MiNiFi-MotionSensor-SMS-Alert+Response.xml and deploy it after updating your custom hostnames and other details details. It looks like below: Flow Explained: a)Receiving Sensor Alert from MiNiFi and send SMS An InputPort receives MotionSensor output from MiNiFi, RouteOnAttribute processor verifies the output and send it to a ControlRate processor only if motion is detected. Control rate processor ensures your phone is not flooded with alerts. putEmail processor is configured to send SMS to my phone. b)Check for ALARM request and send signal to MiNiFi ConsumeIMAP processor checks for new ALARM request in a specified folder in my mailbox, when received, triggers a flowfile. RouteOnContent processor verifies the new mail and route it based on sender and content, feeding it to a PostHTTP processor connecting to listenHTTP on MiNiFi triggering ALARM. Now you its time to try out!! References: Raspberry Pi and PIR Motion Sensor article I followed MiNiFi NiFi My GitHub Article Thanks, Jobin George ... View more

Introduction Recently I was asked how to Monitor and alert while MonitorDiskUsage or MonitorMemory Reporting Task WARNINGS are generated when it exceeds a predefined threshold, I am trying to capture the steps to implement the same. Prerequisites To test this, Make sure HDF-2.x version of NiFi is up an running. You Already have a MonitorDiskUsage/MonitorMemory Reporting task created and configured, set a lower threshold to force generate the alert while testing. 3. Make a note of the Reporting Task uuid to be monitored: Creating a Flow to Monitor Reporting Task. Drop a GenerateFlowFile to execute every 5mins, with 1 byte file to trigger the flow. 2. Drop an UpdateAttribute processor to the canvas with below configuration: NIFI_HOST : <Your NiFi FQDN> NIFI_PORT : <Your NiFi http port> REPORTING-TASK-UUID : <Your MonitorDiskUsage or MonitorMemory controller service uuid> 3. Connect Success relationship of GenerateFlowFile to UpdateAttribute. 4. Drop a InvokeHTTP processor to the canvas, and configure it as below: HTTP Method : GET Remote URL : http://${NIFI_HOST}:${NIFI_PORT}/nifi-api/reporting-tasks/${REPORTING-TASK-UUID} 5. Connect Success relation of UpdateAttribute to InvokeHTTP and auto terminate all relationships of InvokeHTTP processor but Response relationship. 6. Drop a EvaluateJsonPath processor to the canvas with below configuration: 7. Auto terminate All relationship of EvaluateJsonPath processor except Matched relationship and connect InvokeHTTP processor’s Response relation to it. 8. Drop a SplitJson processor to canvas with below Configurtaion. [The reason for splitting json is because the REST call to reporting task gives duplicate json array in the result, which contains details we need.] 9. Connect Matched relationship of EvaluateJsonPath to SplitJson processor. 10. Drop another EvaluateJsonPath processor to the canvas with below configuration: LEVEL : $.bulletin.level MESSAGE : $.bulletin.message SOURCE-NAME : $.bulletin.sourceName TIMESTAMP : $.bulletin.timestamp 11. Add a connection with split relationship from SplitJson to second EvaluateJsonPath processor. Auto-terminate other relationships. 12. Add a ControlRate processor so that only one alert is sent for multiple json arrays with below configuration which passes only 1 flowfile per minute. 13. Add a connection from EvaluateJsonPath processor to ControlRate processor with FlowFile Expiration as 30 sec. Auto-terminate other relations. 14. Finally Drop an PutEmail processor to canvas with below configuration to sent your alerts, update with your SMTP details SMTP Hostname : west.xxxx.yourServer.net SMTP Port : 587 SMTP Username : jgeorge@hortonworks.com SMTP Password : Its_myPassw0rd_updateY0urs SMTP TLS : true From : jgeorge@hortonworks.com To : jgeorge@hortonworks.com Subject : ${SOURCE-NAME} ALERT and message content should look something like below to grab all the values: Message : ${MESSAGE} LEVEL : ${LEVEL} TIMESTAMP : ${TIMESTAMP} SOURCE-NAME : ${SOURCE-NAME} 15. Auto terminate all relationships of PutEmail processor and connect Success relationship of ControlRate processor to it. 16. Now you have your flow ready and can start it to monitor and sent Email Alert for UI Notification thrown by NiFi when Disk or Memory utilization exceeds given threshold. The Flow Would Look like below: Staring the flow, Reporting task and testing it 1. Now lets start the MonitorDiskUsage Reporting task to generate an Alert to test it with below configuration with a lower threshold to force generate the alert. I am monitoring disk on my mac and it’s disk is more than 50% utilized, so I will get this WARNING in the NiFi UI. 2. As soon as this Warning show up and the flow created is running you will get alert in your inbox stating the same, similar as below: 3. This concludes the tutorial for Monitoring your Reporting Tasks with NiFi itself. 4. Too lazy to create the flow???.. Download my template here References NiFi REST API NiFI Expression Language Thanks, Jobin George ... View more

Introduction Using NiFi, data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Spark, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports. In this tutorial we learn to capture NiFi app log from the Sandbox and parse it using Java regex and ingest it to Phoenix via Spark or Directly using NiFi PutSql Processor. Prerequisites 1) Assuming you already have latest version of NiFi-1.x/HDF-2.x downloaded as zip file (HDF and HDP cannot be managed by Ambari on same nodes as of now) on to your HW Sandbox Version 2.5, else execute below after ssh connectivity to sandbox is established: # mkdir /opt/HDF-2.1.1 # cd /opt/HDF-2.1.1 # wget http://public-repo-1.hortonworks.com/HDF/2.1.1.0/nifi-1.1.0.2.1.1.0-2-bin.tar.gz # tar -xvf nifi-1.1.0.2.1.1.0-2-bin.tar.gz 2) Spark, Zeppelin, YARN and HDFS are Installed on your VM and started. 3) Hbase is Installed with phoeix Query Server. 4) Download Compatible version [in our case 1.1.0] of "nifi-spark-receiver" and "nifi-site-to-site-client" to Sandbox in a specific location: # mkdir /opt/spark-receiver # cd /opt/spark-receiver # wget http://central.maven.org/maven2/org/apache/nifi/nifi-spark-receiver/1.1.0/nifi-spark-receiver-1.1.0.jar # wget http://central.maven.org/maven2/org/apache/nifi/nifi-site-to-site-client/1.1.0/nifi-site-to-site-client-1.1.0.jar 5) Make sure Git is installed on the VM: # yum install git -y Configuring and Creating Table in Hbase via Phoenix 1) Make sure Hbase components as well as phoenix query server is started. 2) Make sure Hbase is up and running and out of maintenance mode, below properties are set(if not set it and restart the services): - Enable Phoenix --> Enabled - Enable Authorization --> Off 3) Create Phoenix Table after connecting to phoenix shell (or via Zeppelin): # /usr/hdp/current/phoenix-client/bin/sqlline.py sandbox.hortonworks.com:2181:/hbase-unsecure 4) Execute below in the Phoenix shell to create tables in Hbase: CREATE TABLE NIFI_LOG( UUID VARCHAR NOT NULL, EVENT_DATE VARCHAR, BULLETIN_LEVEL VARCHAR, EVENT_TYPE VARCHAR, CONTENT VARCHAR CONSTRAINT pk PRIMARY KEY(UUID)); CREATE TABLE NIFI_DIRECT( UUID VARCHAR NOT NULL, EVENT_DATE VARCHAR, BULLETIN_LEVEL VARCHAR, EVENT_TYPE VARCHAR, CONTENT VARCHAR CONSTRAINT pk PRIMARY KEY(UUID)); Configuring and Restarting Spark 1) Login to Ambari UI and Navigate to Services --> Spark --> Configs --> Custom spark-defaults and add 2 below properties with given values: spark.driver.extraClassPath = /opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/lib/nifi-framework-api-1.1.0.2.1.1.0-2.jar:/opt/spark-receiver/nifi-site-to-site-client-1.1.0.jar:/opt/spark-receiver/nifi-spark-receiver-1.1.0.jar:/opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/lib/nifi-api-1.1.0.2.1.1.0-2.jar:/opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/lib/bootstrap/nifi-utils-1.1.0.2.1.1.0-2.jar:/opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/work/nar/framework/nifi-framework-nar-1.1.0.2.1.1.0-2.nar-unpacked/META-INF/bundled-dependencies/nifi-client-dto-1.1.0.2.1.1.0-2.jar:/opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/work/nar/framework/nifi-framework-nar-1.1.0.2.1.1.0-2.nar-unpacked/META-INF/bundled-dependencies/httpcore-nio-4.4.5.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar spark.driver.allowMultipleContexts = true 2) Once properties are add, restart Spark. Configuring and Starting NiFi 1) Open nifi.properties for updating configurations: # vi /opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/conf/nifi.properties 2) Change NIFI http port to run on 9090 as default 8080 will conflict with Ambari web UI # web properties # nifi.web.http.port=9090 3) Configure NiFi instance to run site-to site by changing below configuration : add a port say 8055 and set "nifi.remote.input.secure" as "false" # Site to Site properties # nifi.remote.input.socket.port=8055 nifi.remote.input.secure=false 4) Now Start [Restart if already running for configuration change to take effect] NiFi on your Sandbox. # /opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/bin/nifi.sh start 5) Make sure NiFi is up and running by connecting to its Web UI from your browser: http://your-vm-ip:9090/nifi/ Building a Flow in NiFi to fetch and parse nifi-app.log 1) Let us build a small flow on NiFi canvas to read app log generated by NiFi itself to feed to Spark: 2) Drop a "TailFile" Processor to canvas to read lines added to "/opt/HDF-2.1.1/nifi-1.1.0.2.1.1.0-2/logs/nifi-app.log". Auto Terminate relationship Failure. 3) Drop a "SplitText" Processor to canvas to split the log file into separate lines. Auto terminate Original and Failure Relationship for now. Connect TailFile processor to SplitText Processor for Success Relationship. 4) Drop a "ExtractText" Processor to canvas to extract portions of the log content to attributes as below. Connect SplitText processor to ExtractText Processor for splits relationship. - BULLETIN_LEVEL:([A-Z]{4,5}) - CONTENT:(^.*) - EVENT_DATE:([^,]*) - EVENT_TYPE:(?<=\[)(.*?)(?=\]) 5) Drop an OutputPort to the canvas and Name it "spark", Once added, connect "ExtractText" to the port for matched relationship. The Flow would look similar as below: 6) Start the flow on NiFi and notice data is stuck in the connection before the output port "spark" Building Spark application 1) To begin with, lets clone the git repo below: # cd /opt/ # git clone https://github.com/jobinthompu/NiFi-Spark-Feeding-Data-to-Spark-Streaming.git 2) Feel free the inspect Spark application code: # vi /opt/NiFi-Spark-Feeding-Data-to-Spark-Streaming/src/main/Spark+NiFi+Phoenix.sh 3) Now let us go ahead and submit the Spark Application to YARN or can run locally via spark-shell # spark-shell --master yarn --deploy-mode client -i /opt/NiFi-Spark-Feeding-Data-to-Spark-Streaming/src/main/Spark+NiFi+Phoenix.sh OR # spark-shell -i /opt/NiFi-Spark-Feeding-Data-to-Spark-Streaming/src/main/Spark+NiFi+Phoenix.sh 4) Make sure the application is submitted and it prints out statistics. 5) Lets Go ahead and verify that the Application is submitted and started in YARN (you can drill down and see the Application-Master spark UI as well): YARN UI: http://your-vm-ip:8088 Or if you Submit the application locally you can verify that by accessing spark shell application UI: http://sandbox.hortonworks.com:4040/executors/ 6) Lets Go back to the NiFi Web UI, if everything worked fine, the data which was pending on the port 'spark' will be gone as it was consumed by Spark. 7) Now Lets Connect to Phoenix and check out the data populated in tables, you can either use Phoenix sqlline command line or Zeppelin: - via phoenix sqlline # /usr/hdp/current/phoenix-client/bin/sqlline.py localhost:2181:/hbase-unsecure SELECT EVENT_DATE,EVENT_TYPE,BULLETIN_LEVEL FROM NIFI_LOG WHERE BULLETIN_LEVEL='INFO' ORDER BY EVENT_DATE LIMIT 20; - via Zeppelin for better visualization Zeppelin UI: http://your-vm-ip:9995/ Extending NiFi Flow to ingest data directly to Phoenix using PutSql processor 1) Lets go ahead and kill the Spark Application by pressing cntrl+c from command-line: 2) Log back to NiFi UI currently running the flow, and stop the entire flow. 3) Drop a RouteOnAttribute processor to canvas for Matched relation from ExtractText processor and configure it with below property and auto terminate unmatched relation. DEBUG : ${BULLETIN_LEVEL:equals('DEBUG')} ERROR : ${BULLETIN_LEVEL:equals('ERROR')} INFO : ${BULLETIN_LEVEL:equals('INFO')} WARN : ${BULLETIN_LEVEL:equals('WARN')} 4) Drop an AttributesToJSON processor to canvas with below configuration and connect RouteOnAttribute's DEBUG,ERROR,INFO,DEBUG relations to it. Attributes List : uuid,EVENT_DATE,BULLETIN_LEVEL,EVENT_TYPE,CONTENT Destination : flowfile-content 5) Create and enable DBCPConnectionPool with name "Phoenix-Spark" with below configuration: Database Connection URL : jdbc:phoenix:sandbox.hortonworks.com:2181:/hbase-unsecure Database Driver Class Name : org.apache.phoenix.jdbc.PhoenixDriver Database Driver Location(s) : /usr/hdp/current/phoenix-client/phoenix-client.jar 6) Drop a ConvertJSONToSQL to canvas with below configuration, connect AttributesToJSON's success relation to it, auto terminate Failure relation for now after connecting to Phoenix-Spark DB Controller service. 7) Drop a ReplaceText processor canvas to update INSERT statements to UPSERT for Phoenix with below configuration, connect sql relation of ConvertJSONToSQL auto terminate original and Failure relation. 😎 Finally add a PutSQL processor with below configurations and connect it to ReplaceText's success relation and auto terminate all of its relations. 9) The final flow including both ingestion via Spark and direct to phoenix using PutSql is complete, it should look similar to below: 10) Now go ahead and start the flow to ingest data to both Tables via Spark and directly from NiFi. 11) Login back to Zeppelin to see if data is populated in the NIFI_DIRECT table. %jdbc(phoenix) SELECT EVENT_DATE,EVENT_TYPE,BULLETIN_LEVEL FROM NIFI_DIRECT WHERE BULLETIN_LEVEL='INFO' ORDER BY EVENT_DATE Too Lazy to create flow??? download my flow template here This completes the tutorial, You have successfully: - Installed and Configured HDF 2.1 on your HDP-2.5 Sandbox. - Created a Data flow to pull logs and then to Parse it and make it available on a Site-to-site enabled NiFi port. - Created a Spark Application to consume data from NiFi via Site-to-Site and Ingest it to Hbase via Phoenix. - Directly Ingested Data to Phoenix with PutSQL Processor in NiFi with out using Spark. - Viewed the Ingested data from Phoenix command line and Zeppelin References: Mark Payne's - NiFi-Spark My GitHub Article Thanks, Jobin George ... View more

Introduction Recently a customer asked me how to change destination of a connection which still contains data, but the destination is stopped. Using NiFi REST Api we can change the flow, here in this article I am trying to capture steps to update destination of a connection using REST API. The requirement around this was to push incoming data to different flows on a timely manner. Prerequisites 1) To test this, Make sure HDF-2.x version of NiFi is up an running 2) Minimum 3 processors are on the canvas with connection like below: 3) Note the destination's uuid [In my case 'PutNext' processor's uuid] 4) Note the uuid of the connection that has to be made to the 'PutNext' processor 'GET'ing the details of connection 1) Execute the below command on your terminal with uuid of the connection: curl -i -X GET http://localhost:8080/nifi-api/connections/dcbee9dd-0159-1000-45a7-8306c28f2786 2) Now Copy the result of the GET curl command and update the below section with uuid of the 'PutNext' processor. "destination":{"id": "destinationId":" "destinationId": 3) Remove the below json element from the copied result: "permissions":{"canRead":true,"canWrite":true}, Updating Destination with PUT REST calls 1) Once updated, run the PUT REST API call from command line using curl as below: curl -i -X PUT -H 'Content-Type: application/json' -d '{ **MY UPDATED JSON**}' http://localhost:8080/nifi-api/connections/dcbee9dd-0159-1000-45a7-8306c28f2786 My Sample command is as below: curl -i -X PUT -H 'Content-Type: application/json' -d '{ "revision": { "clientId": "dd1c2f03-0159-1000-845b-d5c732a49869", "version": 15 }, "id": "dcbee9dd-0159-1000-45a7-8306c28f2786", "uri": "http://localhost:8080/nifi-api/connections/dcbee9dd-0159-1000-45a7-8306c28f2786", "component": { "id": "dcbee9dd-0159-1000-45a7-8306c28f2786", "parentGroupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "source": { "id": "dcbea89f-0159-1000-278c-cc38bab689bf", "type": "PROCESSOR", "groupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "name": "GenerateFlowFile", "running": false, "comments": "" }, "destination": { "id": "dcbebd86-0159-1000-7559-d77d1e05c910", "type": "PROCESSOR", "groupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "name": "PutFile", "running": false, "comments": "" }, "name": "", "labelIndex": 1, "zIndex": 0, "selectedRelationships": ["success"], "availableRelationships": ["success"], "backPressureObjectThreshold": 10000, "backPressureDataSizeThreshold": "1 GB", "flowFileExpiration": "0 sec", "prioritizers": [], "bends": [] }, "status": { "id": "dcbee9dd-0159-1000-45a7-8306c28f2786", "groupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "name": "success", "statsLastRefreshed": "09:02:22 EST", "sourceId": "dcbea89f-0159-1000-278c-cc38bab689bf", "sourceName": "GenerateFlowFile", "destinationId": "dcbebd86-0159-1000-7559-d77d1e05c910", "destinationName": "PutFile", "aggregateSnapshot": { "id": "dcbee9dd-0159-1000-45a7-8306c28f2786", "groupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "name": "success", "sourceName": "GenerateFlowFile", "destinationName": "PutFile", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesQueued": 18, "bytesQueued": 18, "queued": "18 (18 bytes)", "queuedSize": "18 bytes", "queuedCount": "18" } }, "bends": [], "labelIndex": 1, "zIndex": 0, "sourceId": "dcbea89f-0159-1000-278c-cc38bab689bf", "sourceGroupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "sourceType": "PROCESSOR", "destinationId": "dcbebd86-0159-1000-7559-d77d1e05c910", "destinationGroupId": "cbe6e53b-0158-1000-e36a-f9d26bb1b510", "destinationType": "PROCESSOR" }' http://localhost:8080/nifi-api/connections/dcbee9dd-0159-1000-45a7-8306c28f2786 2) Once you execute the above, if the update is successful, you will get below result: HTTP/1.1 200 OK Date: Fri, 27 Jan 2017 14:59:37 GMT Cache-Control: private, no-cache, no-store, no-transform Content-Type: application/json Transfer-Encoding: chunked Server: Jetty(9.3.9.v20160517) Result will be similar as below: 3) Now login Back to the NiFi UI and make sure the change is done: References: NiFi API My github Thanks, Jobin George ... View more

Introduction Recently I was asked how to monitor and alert flowfile count in a connection queue when it exceeds a predefined threshold, I am trying to capture the steps to implement the same. Prerequisites 1) To test this, Make sure HDF-2.x version of NiFi is up an running. 2) You Already have a connection with data queued in it(say more than 20 flowfiles). Else you can create one like below: 3) Make a note of the Connection name and uuid to be monitored: Creating a Flow to Monitor Connection Queue Count. 1) Drop a GenerateFlowFile processor to the canvas and make "Run Schedule" 60 sec so we dont execute the flow to often. 2) Drop an UpdateAttribute processor, connect GenerateFlowFile's success relation to it and add below properties to it( the connection uuid noted above, threshold say 20, your NiFi host and port): CONNECTION_UUID : dcbee9dd-0159-1000-45a7-8306c28f2786 COUNT : 20 NIFI_HOST : localhost NIFI_PORT : 8080 3) Drop a InvokeHTTP processor to the canvas, connect UpdateAttribute's success relation to it, auto terminate all other relations and update its 2 properties as below: HTTP Method : GET Remote URL : http://${NIFI_HOST}:${NIFI_PORT}/nifi-api/connections/${CONNECTION_UUID} 4) Drop an EvaluateJsonPath processor to extract values from json with below properties, connect Response relation of InvokeHTTP to it, and auto terminate its failure and unmatched relations. QUEUE_NAME : $.status.name QUEUE_SIZE : $.status.aggregateSnapshot.flowFilesQueued 5) Drop a RouteOnAttribute processor to the canvas with below configs, connect EvaluateJsonPath's matched relation to it and auto terminate its unmatched relation. Queue_Size_Exceeded : ${QUEUE_SIZE:gt(${COUNT})} 6) Lastly add a PutEmail processor, connect RouteOnAttribute's matched relation to it and auto terminate all its relations. below are my properties set, you have to update it with your SMTP details: SMTP Hostname : west.xxxx.yourServer.net SMTP Port : 587 SMTP Username : jgeorge@hortonworks.com SMTP Password : Its_myPassw0rd_updateY0urs SMTP TLS : true From : jgeorge@hortonworks.com To : jgeorge@hortonworks.com Subject : Queue Size Exceeded Threshold and message content should look something like below to grab all the values: Message : Queue Size Exceeded Threshold for ${CONNECTION_UUID} Connection Name : ${QUEUE_NAME} Threshold Set : ${COUNT} Current FlowFile Count : ${QUEUE_SIZE} 7) Now the flow is completed and li should look similar to below: Staring the flow and testing it 1) Lets make sure at least 21 flow files are pending in the connection named 'DataToFileSystem' which was created in the Prerequisites 2) Now lets start the flow and you should receive mail alert from NiFi stating the count exceeded Threshold set which is 20 in our case. My sample alert looks like below: 3) This concludes the tutorial for monitoring your connection queue count with NiFi. 4) Too lazy to create the flow???.. Download my template here References NiFi REST API NiFI Expression Language Thanks, Jobin George ... View more

Introduction When the NiFi bootstrap starts or stops NiFi, or detects that it has died unexpectedly, it is able to notify configured recipients. Currently, the only mechanism supplied is to send an e-mail notification. Prerequisite 1) Assuming you already have HDF-2.x Installed, Ambari and NiFi is up and running. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps] Configuring NiFi property files in Ambari 1) To setup email notifications we have to update only two configurations file bootstrap.conf and bootstrap-notification-services.xml 2) We have to update appropriate properties in Ambari to configure it, first we have to edit Template for bootstrap.conf to update below properties. Uncomment below lines in the properties file: nifi.start.notification.services=email-notification nifi.stop.notification.services=email-notification nifi.dead.notification.services=email-notification 3) Edit Template for bootstrap-notification-services.xml and make sure your SMTP settings are updated, and are uncommented. Sample configuration is given below: <service> <id>email-notification</id> <class>org.apache.nifi.bootstrap.notification.email.EmailNotificationService</class> <property name="SMTP Hostname">west.xxxx.server.net</property> <property name="SMTP Port">587</property> <property name="SMTP Username">jgeorge@hortonworks.com</property> <property name="SMTP Password">Th1sisn0tmypassw0rd</property> <property name="SMTP TLS">true</property> <property name="From">jgeorge@hortonworks</property> <property name="To">jgeorge@hortonworks.com</property> </service> 4) Save the Config changes in Ambari after uncommenting the, <service> property, confirm when asked and restart service. Testing NiFi Notification Services 1) Once restarted, you will see both stopped and started alerts in your inbox with details. Stopped Email Alert: Started Email Alert: 2) Try out stopping and killing NiFi process [make sure you don’t kill bootstrap process which monitors NiFi which in turn restarts NiFi process.] Died Email Alert: References NiFi notification_services Thanks, Jobin George ... View more

Introduction ControllerStatusReportingTask Logs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections. By Default, when configured and started it goes directly to nifi-app.log. These can be configured in the NiFi logging configuration to log to different files, here I try to describe steps to log it to a separate log file with Ambari. Prerequisite 1) Assuming you already have HDF-2.x Installed on your VM/Server, Ambari, NiFi is up and running. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps] Configuring "Advanced nifi-node-logback-env" section in Ambari 1. Navigate to your browser window and type in URL for Ambari as below and login to Ambari UI [UI is accessible at port 8080] http://<YOUR_IP>:8080/ 2. Once logged in click on NiFi service option on left side, click on “Configs” and expand “Advanced nifi-node-logback-env” section in configs and edit “logback.xml” template and add below lines just before last line </configuration> <appender name="5MINUTES_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/5minutesStatistics.log</file> <rollingPolicy> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/5minutesStatistics_%d.log</fileNamePattern> <maxHistory>5</maxHistory> </rollingPolicy> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> <logger name="org.apache.nifi.controller.ControllerStatusReportingTask" level="INFO" additivity="false"> <appender-ref ref="5MINUTES_FILE" /> </logger> 3. Once above lines for creating a new file named 5minutesStatistics.log to save all the 5Minutes statistics details the Reporting task creates, click save and enter details of what configuration is changed. 4. Once saved, Ambari will suggest restart of NiFi service, click restart (It might take up to 2 minutes to complete restart and NiFi UI to come online) Configuring ControllerStatusReportingTask in NiFi 1. Once NiFi is restarted, navigate to NiFi User Interface on any node and click on the ‘Controller Settings’ tab on right top corner. A window like below will popup. Select “Reporting Tasks” tab Click on the ‘+’ on right corner, when a selection is requested click on ‘ControllerStatusReportingTask’ 2. Once selected click add, the Reporting task will be in stopped state, Now you may click start: 3. Once started, the 5minutes statistics would have started logging onto 5minutesStatistics.log Verifying Log created in the Server 1. ControllerStatusReportingTask will log the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections to 5minutesStatistics.log. Let’s verify the same: # tail -f /var/log/nifi/5minutesStatistics.log 2. ControllerStatusReportingTask started logging the 5-min processor status to the log we specified and it will be rolled as per the configuration we provided. 3. Once verified, stop the Controller service. Hope this Helps.. Thanks, Jobin George ... View more

Introduction By integrating with LDAP, username/password authentication can be enabled in NiFi. This tutorial provides step by step instructions to setup NiFi - LDAP Authentication via Ambari (Using Knox Demo Ldap Server) Prerequisite 1) Assuming you already have HDF-2.x Installed on your VM/Server, Ambari, NiFi is up and running with out security. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps] Setting up Demo LDAP Server 1) As HDF and HDP cannot co-exist on a single node, lets download knox zip file from apache for this tutorial for easily setting up an ldap server. Execute below steps for the same after establishing ssh connectivity to the VM/Server (name of my host is node1): # ssh node1 # mkdir /opt/knox/ # cd /opt/knox/ # wget http://mirror.cogentco.com/pub/apache/knox/0.11.0/knox-0.11.0.zip # unzip knox-0.11.0.zip # /opt/knox/knox-0.11.0/bin/ldap.sh start 2) Make sure ldap server is started and running on port 33389 on your server # lsof -i:33389 OR # netstat -anp | grep 33389 3) Below credentials are part of knox demo ldap we just started. We can use any of the users to login to NiFi after integration. tom/tom-password admin/admin-password sam/sam-password guest/guest-password Configuring NiFi for LDAP Authentication via Ambari 1. Login to Ambari UI in the server URL, Click on the NiFi service à and then click on Config tab, expand “Advanced nifi-ambari-ssl-config ” section, update configuration as below: Initial Admin Identity : uid=admin,ou=people,dc=hadoop,dc=apache,dc=org Enable SSL? : {click check box} Key password : hadoop Keystore password : hadoop Keystore type : JKS 2. Enter Below as the Truststore and DN configurations : Truststore password : hadoop Truststore type : JKS NiFi CA DN prefix : CN= NiFi CA DN suffix : , OU=NIFI 3. Provide the configuration as below for Node Identities and keystore details: NiFi CA Force Regenerate? : {click check box} NiFi CA Token : hadoop Node Identities : <property name="Node Identity 1">CN=node1, OU=NIFI</property> Tip: Say If I am having a 3 node cluster with node1, node2 and node3 as part f it, above configuration looks like below: <property name="Node Identity 1">CN=node1, OU=NIFI</property> <property name="Node Identity 2">CN=node2, OU=NIFI</property> <property name="Node Identity 3">CN=node3, OU=NIFI</property> 4. In the Ambari UI, choose NiFi service and select config tab. We have to update two set of properties, in the “Advanced nifi-properties ” section update nifi.security.user.login.identity.provider as ldap-provider nifi.security.user.login.identity.provider=ldap-provider 5. Now in the “Advanced nifi-login-identity-providers-env ” section, update the “Template for loginidentity- providers.xml “ property with below configurations just above </loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">uid=admin,ou=people,dc=hadoop,dc=apache,dc=org</property> <property name="Manager Password">admin-password</property> <property name="TLS - Keystore">/usr/hdf/current/nifi/conf/keystore.jks</property> <property name="TLS - Keystore Password">hadoop</property> <property name="TLS - Keystore Type">JKS</property> <property name="TLS - Truststore">/usr/hdf/current/nifi/conf/truststore.jks</property> <property name="TLS - Truststore Password">hadoop</property> <property name="TLS - Truststore Type">JKS</property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://node1:33389</property> <property name="User Search Base">ou=people,dc=hadoop,dc=apache,dc=org</property> <property name="User Search Filter">uid={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> 6. Once All properties are updated, click save and when prompted, click restart. 7. Once restarted, you can try connecting to nifi URL, you should see the below screen, enter credentials as below for admin user the configured Initial Admin Identity and click LOG IN https://node1:9091/nifi/ --> in my case host is node1 admin/admin-password 8. You should be able to login as Admin user for NiFi and should see the below UI: Adding a User and Providing Access to UI 1) Let us go ahead and create a user jobin in ldap so that we can give access for him to NiFi UI. 2) Edit the users.ldif file with below entry in the knox/conf directory and restart the server: # vi /opt/knox/knox-0.11.0/conf/users.ldif Add below entry to the end of the file # entry for sample user jobin dn: uid=jobin,ou=people,dc=hadoop,dc=apache,dc=org objectclass:top objectclass:person objectclass:organizationalPerson objectclass:inetOrgPerson cn: jobin sn: jobin uid: jobin userPassword:jobin-password 3) Once added lets stop and start the ldap server: # /opt/knox/knox-0.11.0/bin/ldap.sh stop # /opt/knox/knox-0.11.0/bin/ldap.sh start 4) While logged in as admin on the nifi UI, Lets us add a user jobin with below id by clicking '+ user' button on top right 'users' menu like below: uid=jobin,ou=people,dc=hadoop,dc=apache,dc=org Enter the above value and click OK. 5. Now close the users window and click to open 'policies' window on the management menu on the top right corner below 'users' menu. click "+user" button on right top corner, on the pop up, enter jobin and select the user and click OK. 6. Once policy added, it will look like below: 7. Now you may log out as admin and provide below credentials to login as 'jobin' user, jobin/jobin-password 8. you should be able to login and view the UI, but wont have privilege to add anything to the canvas. (as jobin is given only read access) you may login back as admin and give required access. This completes the tutorial, You have successfully: - Installed and Configured HDF 2.0 on your server. - Downloaded and started knox Demo Ldap Server - Configured NiFi to use Knox Ldap to Authenticate users where NiFi Initial Admin is from Ldap. - Restarted NiFi and verified access for admin user in NiFi UI. - Created a new user jobin in ldap, added him to NiFi user list and gave read access. - Verified access for user jobin Thanks, Jobin George ... View more

Introduction Using NiFi, data can be exposed in such a way that a receiver can pull from it by adding an Output Port to the root process group. For Storm, we will use this same mechanism - we will use the Site-to-Site protocol to pull data from NiFi's Output Ports. In this tutorial we learn to capture NiFi app log from the Sandbox and parse it using Java regex and ingest it to Phoenix via Storm or Directly using NiFi PutSql Processor. Prerequisites 1) Assuming you already have latest version of NiFi-1.x/HDF-2.x downloaded as zip file (HDF and HDP cannot be managed by Ambari on same nodes as of now) on to your HW Sandbox Version 2.5, else execute below after ssh connectivity to sandbox is established: # cd /opt/ # wget http://public-repo-1.hortonworks.com.s3.amazonaws.com/HDF/centos6/2.x/updates/2.0.1.0/HDF-2.0.1.0-centos6-tars-tarball.tar.gz # tar -xvf HDF-2.0.1.0-12.tar.gz 2) Storm, Zeppelin are Installed on your VM and started. 3) Hbase is Installed with phoeix Query Server. 4) Make sure Maven is installed, if not already, execute below steps: # curl -o /etc/yum.repos.d/epel-apache-maven.repo https://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo # yum -y install apache-maven # mvn -version Configuring and Creating Table in Hbase via Phoenix 1) Make sure Hbase components as well as phoenix query server is started. 2) Make sure Hbase is up and running and out of maintenance mode, below properties are set(if not set it and restart the services): - Enable Phoenix --> Enabled - Enable Authorization --> Off 3) Create Phoenix Table after connecting to phoenix shell (or via Zeppelin): # /usr/hdp/current/phoenix-client/bin/sqlline.py sandbox.hortonworks.com:2181:/hbase-unsecure 4) Execute below in the Phoenix shell to create tables in Hbase: CREATE TABLE NIFI_LOG( UUID VARCHAR NOT NULL, EVENT_DATE DATE, BULLETIN_LEVEL VARCHAR, EVENT_TYPE VARCHAR, CONTENT VARCHAR CONSTRAINT pk PRIMARY KEY(UUID)); CREATE TABLE NIFI_DIRECT( UUID VARCHAR NOT NULL, EVENT_DATE VARCHAR, BULLETIN_LEVEL VARCHAR, EVENT_TYPE VARCHAR, CONTENT VARCHAR CONSTRAINT pk PRIMARY KEY(UUID)); Configuring and Starting NiFi 1) Open nifi.properties for updating configurations: # vi /opt/HDF-2.0.1.0/nifi/conf/nifi.properties 2) Change NIFI http port to run on 9090 as default 8080 will conflict with Ambari web UI # web properties # nifi.web.http.port=9090 3) Configure NiFi instance to run site-to site by changing below configuration : add a port say 8055 and set "nifi.remote.input.secure" as "false" # Site to Site properties # nifi.remote.input.socket.port=8055 nifi.remote.input.secure=false 4) Now Start [Restart if already running for configuration change to take effect] NiFi on your Sandbox. # /opt/HDF-2.0.1.0/nifi/bin/nifi.sh start 5) Make sure NiFi is up and running by connecting to its Web UI from your browser: http://your-vm-ip:9090/nifi/ Building a Flow in NiFi to fetch and parse nifi-app.log 1) Let us build a small flow on NiFi canvas to read app log generated by NiFi itself to feed to Storm: 2) Drop a "TailFile" Processor to canvas to read lines added to "/opt/HDF-2.0.1.0/nifi/logs/nifi-user.log". Auto Terminate relationship Failure. 3) Drop a "SplitText" Processor to canvas to split the log file into separate lines. Auto terminate Original and Failure Relationship for now. Connect TailFile processor to SplitText Processor for Success Relationship. 4) Drop a "ExtractText" Processor to canvas to extract portions of the log content to attributes as below. Connect SplitText processor to ExtractText Processor for splits relationship. - BULLETIN_LEVEL : ([A-Z]{4,5}) - CONTENT : (^.*) - EVENT_DATE : ([^,]*) - EVENT_TYPE : (?<=\[)(.*?)(?=\]) 5) Drop an OutputPort to the canvas and Name it "OUT", Once added, connect "ExtractText" to the port for matched relationship. The Flow would look similar as below: 6) Start the flow on NiFi and notice data is stuck in the connection before the output port "OUT" Building Storm application jar with maven 1) To begin with, lets clone the artifacts, feel free to inspect the dependencies and NiFiStormStreaming.java # cd /opt/ # git clone https://github.com/jobinthompu/NiFi-Storm-Integration.git 2) Feel free the inspect pom.xml to verify the dependencies. # cd /opt/NiFi-Storm-Integration # vi pom.xml 3) Lets rebuild Storm jar with artifacts (this might take several minutes). # mvn package 4) Once the build is SUCCESSFUL, make sure the NiFiStormTopology-Uber.jar is generated in the target folder: # ls -l /opt/NiFi-Storm-Integration/target/NiFiStormTopology-Uber.jar 5) Now let us go ahead and submit the topology in storm (make sure the NiFi flow created above is running before submitting topology). # cd /opt/NiFi-Storm-Integration # storm jar target/NiFiStormTopology-Uber.jar NiFi.NiFiStormStreaming & 6) Lets Go ahead and verify the topology is submitted on the Storm View in Ambari as well as Storm UI: Ambari UI: http://your-vm-ip:8080 Storm UI: http://your-vm-ip:8744/index.html 7) Lets Go back to the NiFi Web UI, if everything worked fine, the data which was pending on the port OUT will be gone as it was consumed by Storm. 😎 Now Lets Connect to Phoenix and check out the data populated in tables, you can either use Phoenix sqlline command line or Zeppelin a) via phoenix sqlline # /usr/hdp/current/phoenix-client/bin/sqlline.py localhost:2181:/hbase-unsecure SELECT EVENT_DATE,EVENT_TYPE,BULLETIN_LEVEL FROM NIFI_DIRECT WHERE BULLETIN_LEVEL='ERROR' ORDER BY EVENT_DATE; b) via Zeppelin for better visualization Zeppelin UI: http://your-vm-ip:9995/ 9) No you can Change the code as needed, re-built the jar and re-submit the topologies. Extending NiFi Flow to ingest data directly to Phoenix using PutSql processor 1) Lets go ahead and kill the storm topology from command-line (or from Ambari Storm-View or Storm UI) # storm kill NiFi-Storm-Phoenix 2) Log back to NiFi UI currently running the flow, and stop the entire flow. 3) Drop a RouteOnAttribute processor to canvas for Matched relation from ExtractText processor and configure it with below property and auto terminate unmatched relation. DEBUG : ${BULLETIN_LEVEL:equals('DEBUG')} ERROR : ${BULLETIN_LEVEL:equals('ERROR')} INFO : ${BULLETIN_LEVEL:equals('INFO')} WARN : ${BULLETIN_LEVEL:equals('WARN')} 4) Drop an AttributesToJSON processor to canvas with below configuration and connect RouteOnAttribute's DEBUG,ERROR,INFO,DEBUG relations to it. Attributes List : uuid,EVENT_DATE,BULLETIN_LEVEL,EVENT_TYPE,CONTENT Destination : flowfile-content 5) Create and enable DBCPConnectionPool with name "Phoenix-Storm" with below configuration: Database Connection URL : jdbc:phoenix:sandbox.hortonworks.com:2181:/hbase-unsecure Database Driver Class Name : org.apache.phoenix.jdbc.PhoenixDriver Database Driver Location(s) : /usr/hdp/current/phoenix-client/phoenix-client.jar 6) Drop a ConvertJSONToSQL to canvas with below configuration, connect AttributesToJSON's success relation to it, auto terminate Failure relation for now after connecting to Phoenix-Storm DB Controller service. 7) Drop a ReplaceText processor canvas to update INSERT statements to UPSERT for Phoenix with below configuration, connect sql relation of ConvertJSONToSQL auto terminate original and Failure relation. 😎 Finally add a PutSQL processor with below configurations and connect it to ReplaceText's success relation and auto terminate all of its relations. 9) The final flow including both ingestion via Storm and direct to phoenix using PutSql is complete, it should look similar to below: 10) Now go ahead and start the flow to ingest data to both Tables via storm and directly from NiFi. 11) Login back to Zeppelin to see if data is populated in the NIFI_DIRECT table. %jdbc(phoenix) SELECT EVENT_DATE,EVENT_TYPE,BULLETIN_LEVEL FROM NIFI_DIRECT WHERE BULLETIN_LEVEL='INFO' ORDER BY EVENT_DATE - Too Lazy to create flow??? download my flow template here. This completes the tutorial, You have successfully: - Installed and Configured HDF 2.0 on your HDP-2.5 Sandbox. - Created a Data flow to pull logs and then to Parse it and make it available on a Site-to-site enabled NiFi port. - Created a Storm topology to consume data from NiFi via Site-to-Site and Ingest it to Hbase via Phoenix. - Directly Ingested Data to Phoenix with PutSQL Processor in NiFi with out using Storm - Viewed the Ingested data from Phoenix command line and Zeppelin References: bbende's - nifi-storm Github Repo Thanks, Jobin George ... View more