1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1856 | 04-03-2024 06:39 AM | |
| 2887 | 01-12-2024 08:19 AM | |
| 1594 | 12-07-2023 01:49 PM | |
| 2360 | 08-02-2023 07:30 AM | |
| 3248 | 03-29-2023 01:22 PM |
08-14-2017
09:27 PM
5 Kudos
Sometimes you want to trigger events with a click on a special touchpad or device mounted somewhere. This could be in a factory, on a door or at your desk. For me it's a small device on my desk that I can use to trigger events. I have this running every 15 seconds looking for gestures. I could put it in an infinite loop, but Python and RPI could leak some memory. We are very constrained, so I am trying to keep it a little more minimal. I keep my batch duration to 15 seconds and my run schedule to 15 seconds. Let's Build A Simple NiFi Flow to Receive The JSON Data and React To It In my RouteOnContent, I just look for the word "center". I have thought of many options for running sql, doing a backup, etc.. Build The MiniFi Flow in NiFi Downloaded minifi 0.2.0 and minifi 0.2.0 toolkit (you can use newer, but make sure you install the same version on the device you are going to move your config.yml to). minifi-toolkit-0.2.0/bin/config.sh transform minififlick.xml config.yml Then SCP that config.yml and the minifi-*.zip to your device. Unzip (or tar-cvf it). Then you can run. This requires Java 8 JDK installed and running on your machine. The Oracle version runs best on RPI. Let's Install and Run MiniFi cd /opt/demo/minifi-0.2.0
bin/minifi.sh install
bin/minifi.sh start Example Message {"flick": "center", "host": "herrflick", "ipaddress": "192.168.1.185", "ts": "2017-08-14 21:19:21", "cputemp": 47.0} The important data is flick which is the gesture made (click, tap, movement, double click, etc...) The other data is one's I always like to grab for devices (hostname, IP Address, timestamp and CPU temperature). Since flick = center, we send a Slack message We could do just about anything you want in the flow based on the trigger. Start backups, send system information, anything you want to trigger on demand. or Source Code: #!/usr/bin/env python
# -*- coding: <utf-8> -*-
# Based on
#Author: Callum Pritchard, Joachim Hummel
#Project Name: Flick 3D Gesture
#Project Description: Sending Flick 3D Gesture sensor data to mqtt
#Version Number: 0.1
#Date: 15/6/17
#Release State: Alpha testing
#Changes: Created
import time
import colorsys
import os
import json
import sys, socket
import subprocess
import time
import datetime
from time import sleep
from time import gmtime, strftime
import signal
import flicklib
import time
from curses import wrapper
some_value = 5000
flicktxt = ''
#### Initialization
# yyyy-mm-dd hh:mm:ss
currenttime= strftime("%Y-%m-%d %H:%M:%S",gmtime())
external_IP_and_port = ('198.41.0.4', 53) # a.root-servers.net
socket_family = socket.AF_INET
host = os.uname()[1]
def getCPUtemperature():
res = os.popen('vcgencmd measure_temp').readline()
return(res.replace("temp=","").replace("'C\n",""))
def IP_address():
try:
s = socket.socket(socket_family, socket.SOCK_DGRAM)
s.connect(external_IP_and_port)
answer = s.getsockname()
s.close()
return answer[0] if answer else None
except socket.error:
return None
def message(publisher, value):
print value
@flicklib.move()
def move(x, y, z):
global xyztxt
xyztxt = '{:5.3f} {:5.3f} {:5.3f}'.format(x,y,z)
@flicklib.flick()
def flick(start,finish):
global flicktxt
flicktxt = 'FLICK-' + start[0].upper() + finish[0].upper()
message('flick',flicktxt)
@flicklib.airwheel()
def spinny(delta):
global some_value
global airwheeltxt
global flicktxt
some_value += delta
if some_value < 0:
some_value = 0
if some_value > 10000:
some_value = 10000
airwheeltxt = str(some_value/100)
flicktxt = airwheeltxt
@flicklib.double_tap()
def doubletap(position):
global doubletaptxt
global flicktxt
doubletaptxt = position
flicktxt = doubletaptxt
@flicklib.tap()
def tap(position):
global taptxt
global flicktxt
taptxt = position
flicktxt = taptxt
@flicklib.touch()
def touch(position):
global touchtxt
global flicktxt
touchtxt = position
flicktxt = touchtxt
def main():
global xyztxt
global flicktxt
global airwheeltxt
global touchtxt
global taptxt
global doubletaptxt
flickcount = 0
airwheeltxt = ''
airwheelcount = 0
touchtxt = ''
touchcount = 0
taptxt = ''
tapcount = 0
doubletaptxt = ''
doubletapcount = 0
time.sleep(0.1)
while flickcount < 100:
if (flicktxt != "") :
flickcount += 100
cpuTemp=int(float(getCPUtemperature()))
ipaddress = IP_address()
row = { 'ts': currenttime, 'host': host, 'cputemp': round(cpuTemp,2), 'ipaddress': ipaddress, 'flick': flicktxt }
json_string = json.dumps(row)
print(json_string)
sys.exit()
time.sleep(0.1)
flickcount += 1
main() See: https://github.com/tspannhw/rpi-rainbowhat References:
https://github.com/PiSupply/Flick.git Apps https://www.pi-supply.com/make/flick-quick-start-faq/ https://github.com/unixweb/Flick https://github.com/unixweb/Flick/blob/master/Flick.py https://github.com/tspannhw/rpi-rainbowhat https://github.com/tspannhw/rpi-rainbowhat/blob/master/minifi.py https://github.com/tspannhw/rpi-sensehat-mqtt-nifi https://github.com/tspannhw/rpi-sensehat-minifi-python https://github.com/tspannhw/rpi-flickhat-minifi/tree/master https://nifi.apache.org/minifi/getting-started.html
... View more
Labels:
08-15-2017
02:39 PM
I just did the default Ambari install and whatever settings were populated then. nifi.provenance.repository.implementation = org.apache.nifi.provenance.PersistentProvenanceRepository The default. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_administration/content/provenance-repository.html https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#provenance-repository
... View more
08-14-2017
11:26 PM
I moved it to the AVRO and made sure the filenames are unique.
... View more
02-06-2018
02:21 PM
Options: You can upgrade. Have non-null messages. Or apply a hotfix. https://github.com/hortonworks/streamline/commit/fdde4fb545b1e3027e4bb7cf364e4fac334bb72c You can contact support to assist with upgrade or hotfix
... View more
08-11-2017
06:17 PM
+1 for an OrcRecordSetWriter
... View more
09-06-2018
01:58 PM
There is a known issue in the PutGCSObject if running behind a secured network and accessing GCS through a proxy. The Proxy Host and Proxy Port added to the PutGCSObject will proxy the object itself, but DOES NOT proxy the GCPCredentialsControllerService, the service that authenticates your connection using yous Service AccountJSON file. At this time, this is a Community Supported Processor, and this issue exists in HFD 3.2 (Nifi v.1.7).
... View more
01-28-2019
11:09 AM
@Greg Keys I am having the same issue with hdf 3.3.1. I have checked the schema file and the input file as well. I have done what was mentioned by @Sriharsha Chintalapani Schema {
"namespace": "hdf.heaptrace.com",
"type": "record",
"name": "PatientField",
"fields": [
{"name": "Patient_name","type": "string"}
]
} JSON data {"Patient_name":"john"}
Please help !!! I have converted data from json to avro and then back again as well using avro tools.
... View more
12-28-2018
01:36 PM
What helped me was, first copying the file to the $NIFI_HOME/lib folder then giving the full path of the jar file in the ExecuteStreamCommand processor. So the config looked like "-jar; /opt/nifi-1.7.1/lib/mycode.jar". Couple things to ensure is that the jar is owned by the same user that NiFi is running as and the jar could be located anywhere as long as you give full path you should be fine.
... View more
08-06-2017
01:40 PM
5 Kudos
Python Word Cloud Integrating existing Python libraries and scripts is very easy in Apache NiFi. I add the library for both version of Python I have on my system, while moving all new scripts to the 3.x branch. Install the library for both Python 2.7 and 3.5 pip install wordcloud
pip3 install wordcloud Example Usage echo "NiFi\nHadoop\nSpark\n" | wordcloud_cli.py --imagefile wordcloud.png For use in NiFi, I wrap my call with a shell script wc.sh echo $1 | tr " " "\n" | wordcloud_cli.py This will build a PNG for me that I can store in a file system or in HDFS, I updated the filename to add png at the end. This will take a parameter to a shell script (our Tweet) and convert it into words usable for a word cloud. You can use other sources or other methods of splitting words. I am pulling twitter messages, so I use ReplaceText to replace the flow file with ${msg} which is just the tweet. Then I execute the Python WordCloud CLI: Example References:
https://amueller.github.io/word_cloud/auto_examples/a_new_hope.html https://amueller.github.io/word_cloud/auto_examples/simple.html#sphx-glr-auto-examples-simple-py https://github.com/amueller/word_cloud
... View more
Labels: