Member since
10-09-2015
86
Posts
179
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
24504 | 12-29-2016 05:19 PM | |
1776 | 12-17-2016 06:05 PM | |
14176 | 08-24-2016 03:08 PM | |
2020 | 07-14-2016 02:35 AM | |
3903 | 07-08-2016 04:29 PM |
07-26-2017
02:35 PM
Thanks @Matt Clarke
... View more
07-19-2017
07:56 PM
Good to know that!!.. thanks for the update @Wynner
... View more
07-19-2017
06:20 PM
@Wynner , Is this a mandatory manual step for NiFi node addition while done via Ambari? I am sure this worked fine with out copying flow.xml.gz in HDF-2.x. Or some work is going on to fix it?
... View more
07-19-2017
06:03 PM
hello @mqureshi, It was a fresh node. there were no flow.xml.gz file present at first, when I started the instance it created a 480b flow.xml.gz file later grow to 593b. I tried deleting it and start/restart NiFi on the fresh node, again a 480b file was created instead of 11880b cluster flow.xml.gz (copying that 11880b cluster flow should have been done by coordinator node but looks like its been done by Ambari - I guess) node4 is the new node and node1 is existing node.
... View more
07-19-2017
05:04 PM
Hello, While trying to add nodes to my existing NiFi cluster with Ambari, I am getting below error in 3.0 version of HDF. It used to work fine in HDF-2.x version. The issue is related to flow differences, but error messages indicates its the issue with AmbariReportingTaskAmbariReport finger print. Ambari used to take care of this in HDF-2.x version. Am I missing something here? Any additional steps which I should be doing in 3.0? I cross checked it with below article for HDF-2.0 by @Matt Clarke, except my cluster is unsecured no other differences: https://community.hortonworks.com/articles/80284/hdf-2x-adding-a-new-nifi-node-to-an-existing-secur.html Any pointers will be appreciated!
2017-07-19 12:39:03,469 ERROR [main] o.a.nifi.controller.StandardFlowService Failed to load flow from cluster due to: org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:936)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:515)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:800)
at org.apache.nifi.NiFi.<init>(NiFi.java:160)
at org.apache.nifi.NiFi.main(NiFi.java:267)
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed configuration is not inheritable by the flow controller because of flow differences: Found difference in Flows:
Local Fingerprint: 7c84501d-d10c-407c-b9f3-1d80e38fe36a3b80ba0f-a6c0-48db-b721-4dbc04cef28eorg.apache.nifi.reporting.ambari.AmbariReportingTaskAmbariReport
Cluster Fingerprint: 7c84501d-d10c-407c-b9f3-1d80e38fe36a428574ae-015d-1000-0000-0000454185d7org.apache.nifi.processors.standard.TailFileNO_VALUEorg.apache.n
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:259)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1576)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:84)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:722)
at org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:911)
... 4 common frames omitted Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
06-09-2017
07:22 PM
hello @Arsalan Siddiqi, Here you can find spark integration with HDF-2.x (still only nifi-1.1) you can figure out the dependency from there.[ first step under:Configuring and Restarting Spark section] https://community.hortonworks.com/content/kbentry/84631/hdf-21-nifi-site-to-site-direct-streaming-to-spark.html Thanks
... View more
05-15-2017
01:37 AM
9 Kudos
Introduction
- Here is a small demo how to flex MiNiFi+NiFi on a Raspberry Pi to detect Motion and send alerts to your phone via SMS, also on your SMS reply it will trigger sound ALARM [Basically to shoo away an intruder]. You can do it from where ever you have cell phone reception.
- Here you can view the screen recording session that Demonstrates how it works! Prerequisite
Raspberry Pi 3, a PIR motion sensor and a speaker connected to it. Details on how to connect PIR Motion Sensor to Raspberry Pi can be found in url under references.
Assuming you already have latest version of HDF/NiFi and Minifi downloaded on your Mac and Pi. Else
Get Latest version of MiNiFi :
# wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-0.1.0-bin.tar.gz
Get Latest version of MiNiFi ToolKit:
# wget http://apache.claz.org/nifi/minifi/0.1.0/minifi-toolkit-0.1.0-bin.tar.gz
Get Latest version of NiFi:
# wget http://apache.claz.org/nifi/1.2.0/nifi-1.2.0-bin.tar.gz
Untar the files and start NiFi on your local machine and MiniFi on your Raspberry Pi
Steps:
Flow on MiNiFi
Download the flow
Pi-MiNiFi-FLow.xml and convert it to YAML format which MiNiFi uses (before you deploy make sure you have your local NiFi URL for RPG rather than what I have in there)
# /root/minifi-toolkit-0.1.0/bin/config.sh transform Pi-MiNiFi-FLow.xml minifi-0.1.0/conf/config.yml
Flow Looks like below:
Flow Explained:
a) Poll for Sensor output and Sent it to NiFi
GenerateFlowFile processor triggers every 5seconds to execute a python script pirtest.py as below using an ExecuteStreamCommand proceesor, result is sent to NiFi running on my local machine via Remote Process Group.
pirtest.py script looks like below:
import RPi.GPIO as GPIO
import time
import os
GPIO.setwarnings(False)
GPIO.setmode(GPIO.BOARD)
GPIO.setup(11, GPIO.IN) #Read output from PIR motion sensor
i=GPIO.input(11)
if i==1: #When output from motion sensor is HIGH
print "Intruder detected",i
b) Play Panic Alarm on SMS trigger from NiFi
listenHTTP processor hosts and listens for any incoming flowfile, when arrived next processor ExecuteStreamCommand executes a python script alarm.py as given below which trigger a panic alarm sound to be played. I used mpg123 to play the alarm sound, you can install it on your raspberry pi using below command:
# sudo apt-get install mpg123
alarm.py script looks like below:
import os
os.system('mpg123 /root/alarm.mp3')
Flow on NiFi
Download the flow MiNiFi-MotionSensor-SMS-Alert+Response.xml and deploy it after updating your custom hostnames and other details details.
It looks like below:
Flow Explained:
a)Receiving Sensor Alert from MiNiFi and send SMS
An InputPort receives MotionSensor output from MiNiFi, RouteOnAttribute processor verifies the output and send it to a ControlRate processor only if motion is detected. Control rate processor ensures your phone is not flooded with alerts. putEmail processor is configured to send SMS to my phone.
b)Check for ALARM request and send signal to MiNiFi
ConsumeIMAP processor checks for new ALARM request in a specified folder in my mailbox, when received, triggers a flowfile. RouteOnContent processor verifies the new mail and route it based on sender and content, feeding it to a PostHTTP processor connecting to listenHTTP on MiNiFi triggering ALARM.
Now you its time to try out!!
References:
Raspberry Pi and PIR Motion Sensor article I followed
MiNiFi
NiFi
My GitHub Article
Thanks,
Jobin George
... View more
Labels:
05-09-2017
06:39 AM
@Bharadwaj Bhimavarapu Not as of today. you need multiple MonitorDiskUsage tasks to do so. But I already did brought this up to product team and hopefully soon we may have that feature.
... View more
05-05-2017
07:33 PM
2 Kudos
Introduction Recently I was asked how to Monitor and alert while MonitorDiskUsage or MonitorMemory Reporting Task WARNINGS are generated when it exceeds a predefined threshold, I am trying to capture the steps to implement the same. Prerequisites
To test this, Make sure HDF-2.x version of NiFi is up an running. You Already have a MonitorDiskUsage/MonitorMemory Reporting task created and configured, set a lower threshold to force generate the alert while testing. 3. Make a note of the Reporting Task uuid to be monitored: Creating a Flow to Monitor Reporting Task.
Drop a GenerateFlowFile to execute every 5mins, with 1 byte file to trigger the flow. 2. Drop an UpdateAttribute processor to the canvas with below configuration: NIFI_HOST : <Your NiFi FQDN>
NIFI_PORT : <Your NiFi http port>
REPORTING-TASK-UUID : <Your MonitorDiskUsage or MonitorMemory controller service uuid> 3. Connect Success relationship of GenerateFlowFile to UpdateAttribute. 4. Drop a InvokeHTTP processor to the canvas, and configure it as below: HTTP Method : GET
Remote URL : http://${NIFI_HOST}:${NIFI_PORT}/nifi-api/reporting-tasks/${REPORTING-TASK-UUID} 5. Connect Success relation of UpdateAttribute to InvokeHTTP and auto terminate all relationships of InvokeHTTP processor but Response relationship. 6. Drop a EvaluateJsonPath processor to the canvas with below configuration: 7. Auto terminate All relationship of EvaluateJsonPath processor except Matched relationship and connect InvokeHTTP processor’s Response relation to it. 8. Drop a SplitJson processor to canvas with below Configurtaion. [The reason for splitting json is because the REST call to reporting task gives duplicate json array in the result, which contains details we need.] 9. Connect Matched relationship of EvaluateJsonPath to SplitJson processor. 10. Drop another EvaluateJsonPath processor to the canvas with below configuration: LEVEL : $.bulletin.level
MESSAGE : $.bulletin.message
SOURCE-NAME : $.bulletin.sourceName
TIMESTAMP : $.bulletin.timestamp 11. Add a connection with split relationship from SplitJson to second EvaluateJsonPath processor. Auto-terminate other relationships. 12. Add a ControlRate processor so that only one alert is sent for multiple json arrays with below configuration which passes only 1 flowfile per minute. 13. Add a connection from EvaluateJsonPath processor to ControlRate processor with FlowFile Expiration as 30 sec. Auto-terminate other relations. 14. Finally Drop an PutEmail processor to canvas with below configuration to sent your alerts, update with your SMTP details SMTP Hostname : west.xxxx.yourServer.net
SMTP Port : 587
SMTP Username : jgeorge@hortonworks.com
SMTP Password : Its_myPassw0rd_updateY0urs
SMTP TLS : true
From : jgeorge@hortonworks.com
To : jgeorge@hortonworks.com
Subject : ${SOURCE-NAME} ALERT and message content should look something like below to grab all the values: Message : ${MESSAGE}
LEVEL : ${LEVEL}
TIMESTAMP : ${TIMESTAMP}
SOURCE-NAME : ${SOURCE-NAME} 15. Auto terminate all relationships of PutEmail processor and connect Success relationship of ControlRate processor to it. 16. Now you have your flow ready and can start it to monitor and sent Email Alert for UI Notification thrown by NiFi when Disk or Memory utilization exceeds given threshold. The Flow Would Look like below: Staring the flow, Reporting task and testing it 1. Now lets start the MonitorDiskUsage Reporting task to generate an Alert to test it with below configuration with a lower threshold to force generate the alert. I am monitoring disk on my mac and it’s disk is more than 50% utilized, so I will get this WARNING in the NiFi UI. 2. As soon as this Warning show up and the flow created is running you will get alert in your inbox stating the same, similar as below: 3. This concludes the tutorial for Monitoring your Reporting Tasks with NiFi itself. 4. Too lazy to create the flow???.. Download my template here References NiFi REST API NiFI Expression Language Thanks, Jobin George
... View more
Labels:
05-03-2017
01:09 PM
Thanks Josh. removing and re-installing zeppelin did the trick. missed to accept the answer.
... View more