1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1915 | 04-03-2024 06:39 AM | |
| 3013 | 01-12-2024 08:19 AM | |
| 1645 | 12-07-2023 01:49 PM | |
| 2421 | 08-02-2023 07:30 AM | |
| 3367 | 03-29-2023 01:22 PM |
08-15-2017
02:39 PM
I just did the default Ambari install and whatever settings were populated then. nifi.provenance.repository.implementation = org.apache.nifi.provenance.PersistentProvenanceRepository The default. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_administration/content/provenance-repository.html https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#provenance-repository
... View more
08-15-2017
02:33 PM
let me check. Nothing was showing up in data provenance in the UI even though the flow was running. it was 4 days behind. After restart it started flowing again.
... View more
08-15-2017
02:22 PM
Restarted NiFi and emptied everything I could and added RAM then it started updating again.
... View more
08-14-2017
11:26 PM
I moved it to the AVRO and made sure the filenames are unique.
... View more
08-14-2017
09:27 PM
5 Kudos
Sometimes you want to trigger events with a click on a special touchpad or device mounted somewhere. This could be in a factory, on a door or at your desk. For me it's a small device on my desk that I can use to trigger events. I have this running every 15 seconds looking for gestures. I could put it in an infinite loop, but Python and RPI could leak some memory. We are very constrained, so I am trying to keep it a little more minimal. I keep my batch duration to 15 seconds and my run schedule to 15 seconds. Let's Build A Simple NiFi Flow to Receive The JSON Data and React To It In my RouteOnContent, I just look for the word "center". I have thought of many options for running sql, doing a backup, etc.. Build The MiniFi Flow in NiFi Downloaded minifi 0.2.0 and minifi 0.2.0 toolkit (you can use newer, but make sure you install the same version on the device you are going to move your config.yml to). minifi-toolkit-0.2.0/bin/config.sh transform minififlick.xml config.yml Then SCP that config.yml and the minifi-*.zip to your device. Unzip (or tar-cvf it). Then you can run. This requires Java 8 JDK installed and running on your machine. The Oracle version runs best on RPI. Let's Install and Run MiniFi cd /opt/demo/minifi-0.2.0
bin/minifi.sh install
bin/minifi.sh start Example Message {"flick": "center", "host": "herrflick", "ipaddress": "192.168.1.185", "ts": "2017-08-14 21:19:21", "cputemp": 47.0} The important data is flick which is the gesture made (click, tap, movement, double click, etc...) The other data is one's I always like to grab for devices (hostname, IP Address, timestamp and CPU temperature). Since flick = center, we send a Slack message We could do just about anything you want in the flow based on the trigger. Start backups, send system information, anything you want to trigger on demand. or Source Code: #!/usr/bin/env python
# -*- coding: <utf-8> -*-
# Based on
#Author: Callum Pritchard, Joachim Hummel
#Project Name: Flick 3D Gesture
#Project Description: Sending Flick 3D Gesture sensor data to mqtt
#Version Number: 0.1
#Date: 15/6/17
#Release State: Alpha testing
#Changes: Created
import time
import colorsys
import os
import json
import sys, socket
import subprocess
import time
import datetime
from time import sleep
from time import gmtime, strftime
import signal
import flicklib
import time
from curses import wrapper
some_value = 5000
flicktxt = ''
#### Initialization
# yyyy-mm-dd hh:mm:ss
currenttime= strftime("%Y-%m-%d %H:%M:%S",gmtime())
external_IP_and_port = ('198.41.0.4', 53) # a.root-servers.net
socket_family = socket.AF_INET
host = os.uname()[1]
def getCPUtemperature():
res = os.popen('vcgencmd measure_temp').readline()
return(res.replace("temp=","").replace("'C\n",""))
def IP_address():
try:
s = socket.socket(socket_family, socket.SOCK_DGRAM)
s.connect(external_IP_and_port)
answer = s.getsockname()
s.close()
return answer[0] if answer else None
except socket.error:
return None
def message(publisher, value):
print value
@flicklib.move()
def move(x, y, z):
global xyztxt
xyztxt = '{:5.3f} {:5.3f} {:5.3f}'.format(x,y,z)
@flicklib.flick()
def flick(start,finish):
global flicktxt
flicktxt = 'FLICK-' + start[0].upper() + finish[0].upper()
message('flick',flicktxt)
@flicklib.airwheel()
def spinny(delta):
global some_value
global airwheeltxt
global flicktxt
some_value += delta
if some_value < 0:
some_value = 0
if some_value > 10000:
some_value = 10000
airwheeltxt = str(some_value/100)
flicktxt = airwheeltxt
@flicklib.double_tap()
def doubletap(position):
global doubletaptxt
global flicktxt
doubletaptxt = position
flicktxt = doubletaptxt
@flicklib.tap()
def tap(position):
global taptxt
global flicktxt
taptxt = position
flicktxt = taptxt
@flicklib.touch()
def touch(position):
global touchtxt
global flicktxt
touchtxt = position
flicktxt = touchtxt
def main():
global xyztxt
global flicktxt
global airwheeltxt
global touchtxt
global taptxt
global doubletaptxt
flickcount = 0
airwheeltxt = ''
airwheelcount = 0
touchtxt = ''
touchcount = 0
taptxt = ''
tapcount = 0
doubletaptxt = ''
doubletapcount = 0
time.sleep(0.1)
while flickcount < 100:
if (flicktxt != "") :
flickcount += 100
cpuTemp=int(float(getCPUtemperature()))
ipaddress = IP_address()
row = { 'ts': currenttime, 'host': host, 'cputemp': round(cpuTemp,2), 'ipaddress': ipaddress, 'flick': flicktxt }
json_string = json.dumps(row)
print(json_string)
sys.exit()
time.sleep(0.1)
flickcount += 1
main() See: https://github.com/tspannhw/rpi-rainbowhat References:
https://github.com/PiSupply/Flick.git Apps https://www.pi-supply.com/make/flick-quick-start-faq/ https://github.com/unixweb/Flick https://github.com/unixweb/Flick/blob/master/Flick.py https://github.com/tspannhw/rpi-rainbowhat https://github.com/tspannhw/rpi-rainbowhat/blob/master/minifi.py https://github.com/tspannhw/rpi-sensehat-mqtt-nifi https://github.com/tspannhw/rpi-sensehat-minifi-python https://github.com/tspannhw/rpi-flickhat-minifi/tree/master https://nifi.apache.org/minifi/getting-started.html
... View more
Labels:
08-14-2017
08:47 PM
With no login on. I haven't seen any updates to data provenance in 3 days. Powered by Apache NiFi - Version 1.2.0.3.0.0.0-453 06/08/2017 19:02:03 UTC Tagged nifi-1.2.0-RC2 From a1b7710 on branch UNKNOWN
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
08-13-2017
09:20 PM
When I fetch 49 records at a time, 49 flow files converted to ORC go into the putHDFS processor but the result is one small file in HDFS. When I do a select count(*) there is only 1 record in the external table. There are no failed flow files coming out of the HDFS processor. I included a mergecontent processor after the convert to orc and before put to HDFS, a bigger file of merged content is put into HDFS but that results in an error(see below) when I select count(*) from the external table. I suspect the merged file is corrupted, but putting the original 49 files into HDFS doesn’t appear to be working properly either. The External table is simply defined with fields STORED as ORC LOCATION ‘hdfspath’. I tried the DDL with and without orc.compress=’SNAPPY’, even removed all compression from any processor and the results were the same as the mergedcontent file throws the below error every time when selecting from the external table. Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex_1502140036873_0283_2_00, diagnostics=[Task failed, taskId=task_1502140036873_0283_2_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero). Caused by: java.lang.RuntimeException: java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
... View more
Labels:
- Labels:
-
Apache NiFi
08-11-2017
09:12 PM
Unable to get offset lags for kafka. Reason: org.apache.kafka.shaded.common.errors.InvalidTopicException: Topic '[traffic]' is invalid Valid topic that exists. [root@princeton10 bin]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper princeton10.field.hortonworks.com:2181 --list
__consumer_offsets
ambari_kafka_service_check
gps3
gps4
traffic ava.lang.NullPointerException: null value in entry: id=null at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33) at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:135) at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:206) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.convertValue(AvroStreamsSnapshotDeserializer.java:68) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:49) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroStreamsSnapshotDeserializer.doDeserialize(AvroStreamsSnapshotDeserializer.java:39) at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:149) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroKafkaSpoutTranslator.apply(AvroKafkaSpoutTranslator.java:61) at org.apache.storm.kafka.spout.KafkaSpout.emitTupleIfNotEmitted(KafkaSpout.java:335) at org.apache.storm.kafka.spout.KafkaSpout.emit(KafkaSpout.java:316) at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:236) at org.apache.storm.daemon.executor$fn__10364$fn__10379$fn__10410.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Kafka