Introduction - Here is a small demo how to flex MiNiFi+NiFi on a Raspberry Pi to detect Motion and send alerts to your phone via SMS, also on your SMS reply it will trigger sound ALARM [Basically to shoo away an intruder]. You can do it from where ever you have cell phone reception. - Here you can view the screen recording session that Demonstrates how it works! Prerequisite Raspberry Pi 3, a PIR motion sensor and a speaker connected to it. Details on how to connect PIR Motion Sensor to Raspberry Pi can be found in url under references. Assuming you already have latest version of HDF/NiFi and Minifi downloaded on your Mac and Pi. Else Get Latest version of MiNiFi : # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-0.1.0-bin.tar.gz Get Latest version of MiNiFi ToolKit: # wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-toolkit-0.1.0-bin.tar.gz Get Latest version of NiFi: # wget http://apache.claz.org/nifi/1.2.0/nifi-1.2.0-bin.tar.gz Untar the files and start NiFi on your local machine and MiniFi on your Raspberry Pi Steps: Flow on MiNiFi Download the flow Pi-MiNiFi-FLow.xml and convert it to YAML format which MiNiFi uses (before you deploy make sure you have your local NiFi URL for RPG rather than what I have in there) # /root/minifi-toolkit-0.1.0/bin/config.sh transform Pi-MiNiFi-FLow.xml minifi-0.1.0/conf/config.yml Flow Looks like below: Flow Explained: a) Poll for Sensor output and Sent it to NiFi GenerateFlowFile processor triggers every 5seconds to execute a python script pirtest.py as below using an ExecuteStreamCommand proceesor, result is sent to NiFi running on my local machine via Remote Process Group. pirtest.py script looks like below: import RPi.GPIO as GPIO import time import os GPIO.setwarnings(False) GPIO.setmode(GPIO.BOARD) GPIO.setup(11, GPIO.IN) #Read output from PIR motion sensor i=GPIO.input(11) if i==1: #When output from motion sensor is HIGH print "Intruder detected",i b) Play Panic Alarm on SMS trigger from NiFi listenHTTP processor hosts and listens for any incoming flowfile, when arrived next processor ExecuteStreamCommand executes a python script alarm.py as given below which trigger a panic alarm sound to be played. I used mpg123 to play the alarm sound, you can install it on your raspberry pi using below command: # sudo apt-get install mpg123 alarm.py script looks like below: import os os.system('mpg123 /root/alarm.mp3') Flow on NiFi Download the flow MiNiFi-MotionSensor-SMS-Alert+Response.xml and deploy it after updating your custom hostnames and other details details. It looks like below: Flow Explained: a)Receiving Sensor Alert from MiNiFi and send SMS An InputPort receives MotionSensor output from MiNiFi, RouteOnAttribute processor verifies the output and send it to a ControlRate processor only if motion is detected. Control rate processor ensures your phone is not flooded with alerts. putEmail processor is configured to send SMS to my phone. b)Check for ALARM request and send signal to MiNiFi ConsumeIMAP processor checks for new ALARM request in a specified folder in my mailbox, when received, triggers a flowfile. RouteOnContent processor verifies the new mail and route it based on sender and content, feeding it to a PostHTTP processor connecting to listenHTTP on MiNiFi triggering ALARM. Now you its time to try out!! References: Raspberry Pi and PIR Motion Sensor article I followed MiNiFi NiFi My GitHub Article Thanks, Jobin George ... View more

Introduction Recently I was asked how to Monitor and alert while MonitorDiskUsage or MonitorMemory Reporting Task WARNINGS are generated when it exceeds a predefined threshold, I am trying to capture the steps to implement the same. Prerequisites To test this, Make sure HDF-2.x version of NiFi is up an running. You Already have a MonitorDiskUsage/MonitorMemory Reporting task created and configured, set a lower threshold to force generate the alert while testing. 3. Make a note of the Reporting Task uuid to be monitored: Creating a Flow to Monitor Reporting Task. Drop a GenerateFlowFile to execute every 5mins, with 1 byte file to trigger the flow. 2. Drop an UpdateAttribute processor to the canvas with below configuration: NIFI_HOST : <Your NiFi FQDN> NIFI_PORT : <Your NiFi http port> REPORTING-TASK-UUID : <Your MonitorDiskUsage or MonitorMemory controller service uuid> 3. Connect Success relationship of GenerateFlowFile to UpdateAttribute. 4. Drop a InvokeHTTP processor to the canvas, and configure it as below: HTTP Method : GET Remote URL : http://${NIFI_HOST}:${NIFI_PORT}/nifi-api/reporting-tasks/${REPORTING-TASK-UUID} 5. Connect Success relation of UpdateAttribute to InvokeHTTP and auto terminate all relationships of InvokeHTTP processor but Response relationship. 6. Drop a EvaluateJsonPath processor to the canvas with below configuration: 7. Auto terminate All relationship of EvaluateJsonPath processor except Matched relationship and connect InvokeHTTP processor’s Response relation to it. 8. Drop a SplitJson processor to canvas with below Configurtaion. [The reason for splitting json is because the REST call to reporting task gives duplicate json array in the result, which contains details we need.] 9. Connect Matched relationship of EvaluateJsonPath to SplitJson processor. 10. Drop another EvaluateJsonPath processor to the canvas with below configuration: LEVEL : $.bulletin.level MESSAGE : $.bulletin.message SOURCE-NAME : $.bulletin.sourceName TIMESTAMP : $.bulletin.timestamp 11. Add a connection with split relationship from SplitJson to second EvaluateJsonPath processor. Auto-terminate other relationships. 12. Add a ControlRate processor so that only one alert is sent for multiple json arrays with below configuration which passes only 1 flowfile per minute. 13. Add a connection from EvaluateJsonPath processor to ControlRate processor with FlowFile Expiration as 30 sec. Auto-terminate other relations. 14. Finally Drop an PutEmail processor to canvas with below configuration to sent your alerts, update with your SMTP details SMTP Hostname : west.xxxx.yourServer.net SMTP Port : 587 SMTP Username : jgeorge@hortonworks.com SMTP Password : Its_myPassw0rd_updateY0urs SMTP TLS : true From : jgeorge@hortonworks.com To : jgeorge@hortonworks.com Subject : ${SOURCE-NAME} ALERT and message content should look something like below to grab all the values: Message : ${MESSAGE} LEVEL : ${LEVEL} TIMESTAMP : ${TIMESTAMP} SOURCE-NAME : ${SOURCE-NAME} 15. Auto terminate all relationships of PutEmail processor and connect Success relationship of ControlRate processor to it. 16. Now you have your flow ready and can start it to monitor and sent Email Alert for UI Notification thrown by NiFi when Disk or Memory utilization exceeds given threshold. The Flow Would Look like below: Staring the flow, Reporting task and testing it 1. Now lets start the MonitorDiskUsage Reporting task to generate an Alert to test it with below configuration with a lower threshold to force generate the alert. I am monitoring disk on my mac and it’s disk is more than 50% utilized, so I will get this WARNING in the NiFi UI. 2. As soon as this Warning show up and the flow created is running you will get alert in your inbox stating the same, similar as below: 3. This concludes the tutorial for Monitoring your Reporting Tasks with NiFi itself. 4. Too lazy to create the flow???.. Download my template here References NiFi REST API NiFI Expression Language Thanks, Jobin George ... View more