' The Onion Omega 2+ is a small IoT device that runs a simple busybox Linux and can run Micropython. This let's you run some simple application and interact with some sensors and an OLED. Onion Omega 2+ Stats 580MHz Cpu 128MB memory 32mb storage added usb 32gb usb 2 microusb b/g/n wifi 15 gpio 2 pwm 2 uart 1 i2c 1 spi 1 i2s Setting Up the Omega opkg install python-pip pip install --upgrade setuptools pip install paho-mqtt opkg install pyOledExp Upgrading pyOledExp on root from 0.4-1 to 0.5-1... Downloading http://repo.onion.io/omega2/packages/onion/pyOledExp_0.5-1_mipsel_24kc.ipk Configuring pyOledExp. mkdir /mnt/sda1 mount /dev/sda1 /mnt/sda1 ./run.sh > Initializing display > Setting display to ON > Enabling horizontal scrolling to the left > Writing '[{"ipaddress": "192.168.1.176", "endtime": "2018-01-29 00:50:39", "end": "1517187039.44"}]' to display 0 crontab -e crontab -l */1 * * * * /opt/demo/run.sh 1517187305: New connection from 192.168.1.176 on port 1883. 1517187305: New client connected from 192.168.1.176 as onion (c1, k60). 1517187305: Client onion disconnected. BusyBox v1.26.2 () built-in shell (ash) ____ _ ____ / __ \___ (_)__ ___ / __ \__ _ ___ ___ ____ _ / /_/ / _ \/ / _ \/ _ \ / /_/ / ' \/ -_) _ `/ _ `/ \____/_//_/_/\___/_//_/ \____/_/_/_/\__/\_, /\_,_/ W H A T W I L L Y O U I N V E N T ? /___/ ----------------------------------------------------- Ω-ware: 0.1.10 b160 ----------------------------------------------------- poweroff Attributes Related to MQTT Message Sent Example Flow File containing JSON Apache NiFi Flow File to Process Running MQTT on an Mac /usr/local/Cellar/mosquitto/1.4.14_2/sbin/mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf 1517180449: mosquitto version 1.4.14 (build date 2017-10-22 16:34:22+0100) starting 1517180449: Config loaded from /usr/local/etc/mosquitto/mosquitto.conf. 1517180449: Opening ipv6 listen socket on port 1883. 1517180449: Opening ipv4 listen socket on port 1883. 1517180698: New connection from 127.0.0.1 on port 1883. 1517180698: New client connected from 127.0.0.1 as nififorthemqttguy (c1, k60). In our simple example we are just reading the time and IP Address of the device and format it in JSON to send as MQTT messages to an MQTT server read by Apache NiFi. This is a good framework to start with on tiny devices. With the Onion platform you can add GPS, sensors, USB devices, USB webcam and other inputs. These can easily be added to the Python script to send to Apache NiFi as JSON. Source Code https://github.com/tspannhw/onionomega-mqtt-micropython Python Script from OmegaExpansion import oledExp import paho.mqtt.client as client import time import os import datetime import math import random, string import json import sys import socket import json from time import sleep from string import Template from time import gmtime, strftime # Time start = time.time() currenttime= strftime("%Y-%m-%d %H:%M:%S",gmtime()) host = os.uname()[1] external_IP_and_port = ('198.41.0.4', 53) # a.root-servers.net socket_family = socket.AF_INET def IP_address(): try: s = socket.socket(socket_family, socket.SOCK_DGRAM) s.connect(external_IP_and_port) answer = s.getsockname() s.close() return answer[0] if answer else None except socket.error: return None ipaddress = IP_address() status = oledExp.driverInit() status = oledExp.setDisplayPower(1) status = oledExp.scroll (0, 0, 0, 8-1); endtime= strftime("%Y-%m-%d %H:%M:%S",gmtime()) end = time.time() row = [ { 'end': str(end), 'endtime': str(endtime), 'ipaddress': str(ipaddress) } ] json_string = json.dumps(row) broker="192.168.1.193" port=1883 client1= client.Client("onion") #create client object client1.connect(broker,port) #establish connection ret= client1.publish("omega",json_string) client1.disconnect() status = oledExp.write(json_string) print(status) References https://community.hortonworks.com/articles/89455/ingesting-gps-data-from-onion-omega2-devices-with.html https://github.com/tspannhw/onionomega-mqtt-micropython https://github.com/mccollam/omega https://github.com/micropython/micropython-lib/tree/master/umqtt.simple https://docs.onion.io/omega2-docs/using-oled-expansion.html#using-the-libraries-2 https://iotbytes.wordpress.com/paho-mqtt-with-python/ https://www.kickstarter.com/projects/onion/omega2-5-iot-computer-with-wi-fi-powered-by-linux ... View more

The Matrix Creator is an interesting multiple sensor hat that fits on a Raspberry Pi 3. First step is to connect it, which is a simple snap, no soldering required. The specs are pretty impressive: Xilinx Spartan 6 XC6SLX4 FPGA Amtel Cortex-M3 ATSAM3S2 Microcontroller 8 MEMS MP34DB02 audio sensor digital microphones ST LSM9DS1 3D accelerometer, 3D gyroscope, 3d magnetometer IMU ST HTS221 capacitive digital sensor for relative humidity and temperature NXP MPL3115A2 precision pressure sensor with altimetry Silicon Labs EM358X - 2.4 GHz IEEE 802.15.4 Zigbee Sigma Designs ZM5202 - 868/908/921 MHz ZWave Vishay TSOP573 - carrier 38.0 kHz IR Receiver Vishay VEML6070 UV light sensor NXP PN512 NFC reader Everloop 35 RGBW LEDS It runs on Raspian lite and installs via: curl https://matrix-io.github.io/matrix-documentation/install.sh | sh Our Apache NiFi Flow For Processing the Three Types of Data Our Versioned Apache NiFi and MiniFi Flows We tail the three files produced by the three example Python sensors readers Both our MiniFi and Apache NiFi flows are very simple and documented above. Tail data from files as Python writes it, send from MiniFi to Apache NiFi which separate the files into different flows for future processing. We could create schemas, convert to JSON, merge the feeds with JSON, store them in three different data stores or more depending on what you want to do. This can be one on the edge or in Apache NiFi on a cluster. You could have MiniFi or NiFi trigger off specific values or ranges as they need arises. Or like me, you can just store it for later use in your endless HDFS Data Lake. Using Three Existing Examples Getting Temperature, UV and IMU Values. python /home/pi/matrix-creator-malos/src/python_test/test_humidity.py nohup ./humidity.sh & fh = open("/opt/demo/logs/humidity.log", "a") fh.writelines('{0}'.format(humidity_info)) fh.close python /home/pi/matrix-creator-malos/src/python_test/test_uv.py /opt/demo/logs/uv.log python /home/pi/matrix-creator-malos/src/python_test/test_imu.py /opt/demo/logs/imu.log /Volumes/seagate/Apps/minifi-toolkit-0.3.0/bin/config.sh transform $1 config.yml scp config.yml pi@192.168.1.197:/opt/demo/minifi-0.3.0/conf Example Data imu.2603753-2604002.log yaw: 141.655654907 roll: 1.66126561165 accel_x: -0.0261840820312 accel_y: 0.0283813476562 accel_z: 0.978576660156 gyro_x: -0.0672912597656 gyro_y: 2.06359863281 gyro_z: 1.33087158203 mag_x: 0.23982000351 mag_y: 0.189700007439 mag_z: -0.480480015278 uv.172512-172528.log oms_risk: "Low" humidity.29015-29074.log temperature: 21.9526348114 temperature_is_calibrated: true References: https://medium.com/kkbankol-events/raspberry-pi-15662c3ca881 https://creator.matrix.one/#!/examples https://github.com/matrix-io/matrix-creator-malos/blob/master/docs/pressure.md http://community.matrix.one/t/how-to-record-with-pyaudio/357 https://matrix-io.github.io/matrix-documentation/matrix-core/examples/pytests/ https://github.com/matrix-io/matrix-creator-alexa-voice-services https://matrix-io.github.io/matrix-documentation/matrix-hal/getting-started/installation/ https://matrix-io.github.io/matrix-documentation/matrix-core/examples/pytests/ https://matrix-io.github.io/matrix-documentation/setup/ https://www.matrix.one/products/creator ... View more

So I found another low-end affordable platform from China for running MiniFi goodness. For this 512MB RAM machine, I decided to use MiniFi CPP. git clone https://github.com/apache/nifi-minifi-cpp.git apt-get install cmake gcc g++ bison flex libcurl-dev librocksdb-dev librocksdb4.1 uuid-dev uuid libboost-all-dev libssl-dev libbz2-dev liblzma-dev doxygen -y apt-get install -y libleveldb-dev apt-get install -y libxml2 apt-get install libpython3-dev -y apt-get install liblua5.1-0-dev -y apt-get install libusb-1.0.0-0-dev libpng12-dev -y apt-get install docker.io python-virtualenv -y apt-get install libpython3-dev -y apt-get install libgps-dev -y apt-get install libpcap-dev -y apt-get install cmake gcc g++ bison flex -y ./bootstrap.sh interactive UI cd nifi-minifi-cpp-0.3.0-source/ mkdir build cd build cmake .. make make package apt-get install libssl-dev -- The following features have been enabled: * EXPRESSION LANGUAGE EXTENSIONS , This enables NiFi expression language * HTTP CURL , This enables RESTProtocol, InvokeHTTP, and the HTTPClient for Site to Site * ROCKSDB REPOS , This Enables persistent provenance, flowfile, and content repositories using RocksDB * ARCHIVE EXTENSIONS , This Enables libarchive functionality including MergeContent, CompressContent, (Un)FocusArchiveEntry and ManipulateArchive. * SCRIPTING EXTENSIONS , This enables scripting -- The following OPTIONAL packages have been found: * LibRt * Git * BZip2 * LibLZMA * EXPAT * Boost * Doxygen -- The following REQUIRED packages have been found: * BISON * FLEX * CURL * Threads * PythonLibs * ZLIB * UUID * OpenSSL -- The following features have been disabled: * CIVETWEB , This enables ListenHTTP -- The following OPTIONAL packages have not been found: * WinSock * RocksDB * LibArchive * Nettle * LibXml2 This will take some library installs for development and various libraries needed for networking, security and devices. From the interactive UI, I selected to add most of the goodies except a USB camera since I don't have one on this tiny machine. The CPP is different (and much smaller) than the Java one. For first, you have a specific set of processors, some of which are pretty cool like ones for USB Camera image ingestion, TensorFlow processing and other device goodness. You can browse the list in the PROCESSORs link below. Since I built mine from the git clone of the master, I am running the 0.4.0 branch. I could not install RocksDB on OrangePi There's some cool stuff for reporting status. root@orangepizero:/opt/demo/nifi-minifi-cpp-0.4.0# bin/minifi minifi minificontroller minifi.sh root@orangepizero:/opt/demo/nifi-minifi-cpp-0.4.0# bin/minifi.sh start Starting MiNiFi with PID 15831 and pid file /opt/demo/nifi-minifi-cpp-0.4.0/bin/.minifi.pid root@orangepizero:/opt/demo/nifi-minifi-cpp-0.4.0# [2018-01-infol 16:24:17.591] [main] [info] Using MINIFI_HOME=/opt/demo/nifi-minifi-cpp-0.4.0 from environment. [2018-01-infol 16:24:17.592] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi-log.properties [2018-01-26 16:24:17.893] [main] [info] Loading FlowController [2018-01-26 16:24:17.893] [org::apache::nifi::minifi::FlowController] [info] Load Flow Controller from file /opt/demo/nifi-minifi-cpp-0.4.0/conf/config.yml [2018-01-26 16:24:17.895] [org::apache::nifi::minifi::FlowController] [info] Loaded root processor Group [2018-01-26 16:24:17.895] [org::apache::nifi::minifi::FlowController] [info] Initializing timers [2018-01-26 16:24:17.896] [org::apache::nifi::minifi::FlowController] [info] Loaded controller service provider [2018-01-26 16:24:17.896] [org::apache::nifi::minifi::FlowController] [info] Loaded flow repository [2018-01-26 16:24:17.896] [org::apache::nifi::minifi::FlowController] [info] Starting Flow Controller [2018-01-26 16:24:17.898] [org::apache::nifi::minifi::core::controller::StandardControllerServiceProvider] [info] Enabling % controller services [2018-01-26 16:24:17.899] [org::apache::nifi::minifi::c2::C2Agent] [info] Class is RESTSender [2018-01-26 16:24:17.902] [org::apache::nifi::minifi::io::Socket] [error] Could not bind to socket [2018-01-26 16:24:17.903] [org::apache::nifi::minifi::FlowController] [info] Started Flow Controller [2018-01-26 16:24:17.903] [main] [info] MiNiFi started root@orangepizero:/opt/demo/nifi-minifi-cpp-0.4.0/bin# ./minificontroller --list components [2018-01-infol 16:25:16.461] [controller] [info] MINIFI_HOME is not set; determining based on environment. [2018-01-infol 16:25:16.462] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi.properties [2018-01-infol 16:25:16.463] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi-log.properties Components: FlowController root@orangepizero:/opt/demo/nifi-minifi-cpp-0.4.0/bin# ./minificontroller --list connections [2018-01-infol 16:25:32.850] [controller] [info] MINIFI_HOME is not set; determining based on environment. [2018-01-infol 16:25:32.851] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi.properties [2018-01-infol 16:25:32.852] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi-log.properties Connection Names: ./minificontroller --updateflow "config yml" root@orangepizero:/opt/demo/nifi-minifi-cpp-0.4.0/bin# ./minificontroller --getfull [2018-01-infol 16:26:13.296] [controller] [info] MINIFI_HOME is not set; determining based on environment. [2018-01-infol 16:26:13.297] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi.properties [2018-01-infol 16:26:13.298] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /opt/demo/nifi-minifi-cpp-0.4.0/conf/minifi-log.properties 0 are full References https://github.com/apache/nifi-minifi-cpp https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal https://github.com/apache/nifi-minifi-cpp/blob/master/examples/BidirectionalSiteToSite/README.md https://nifi.apache.org/minifi/getting-started.html https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md# https://github.com/apache/nifi-minifi-cpp/blob/master/EXPRESSIONS.md https://cwiki.apache.org/confluence/display/MINIFI/Release+Notes#ReleaseNotes-Versioncpp-0.3.0 https://github.com/apache/nifi-minifi-cpp/blob/master/Extensions.md To Customize C++ Builds https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74685143 Build Extensions with MiniFi C++ https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74685988 MiniFi C++ System properties https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70256416 ... View more

Using SiteToSiteProvenanceReportingTask to Send Provenance to Apache NiFi for Processing. Eating our own provenance food! It's almost comically easy to do this. You set up a task on the server you are reporting on that sends the data to your receiver. That other server you make a simple flow to ingest and process that. I stored it to HBase as JSON as it's a good place to put a lot of data fast. Send The Data You need to create a SiteToSiteProvenanceReportingTask in Controller Settings - Reporting Tasks. It's pretty simple. Set the values above with your destination NiFi server and a port name that you have created already. Receive the Data and Process An Individual JSON Record Split the JSON into Records $.[*] Save to HBase (PutHBaseJSON) First I have to create a table. hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.1.2.2.6.2.0-205, r5210d2ed88d7e241646beab51e9ac147a973bdcc, Sat Aug 26 09:33:50 UTC 2017 hbase(main):001:0> create 'PROVENANCE', 'event' 0 row(s) in 2.9900 seconds => Hbase::Table - PROVENANCE scan 'PROVENANCE' ff91e204-05b0-48aa-a666-7942e3f109ab column=event:previousAttributes, timestamp=1517159115042, value={"path":"./","filename":"humidity.583225-583284.log","s2s.address":"192.168.1.197:55032","s2s.host":"1 92.168.1.197","mime.type":"text/plain","uuid":"9006a1bb-d755-4272-b8d3-76e666c2a7c6","tailfile.original.path":"/opt/demo/logs/humidity.log"} ff91e204-05b0-48aa-a666-7942e3f109ab column=event:previousContentURI, timestamp=1517159115042, value=http://192.168.1.193:8080/nifi-api/provenance-events/61825/content/input ff91e204-05b0-48aa-a666-7942e3f109ab column=event:previousEntitySize, timestamp=1517159115042, value=59 ff91e204-05b0-48aa-a666-7942e3f109ab column=event:processGroupId, timestamp=1517159115042, value=01611005-4e82-1491-ae5d-ca64f59491cb ff91e204-05b0-48aa-a666-7942e3f109ab column=event:processGroupName, timestamp=1517159115042, value=Process MiniFi Creator ff91e204-05b0-48aa-a666-7942e3f109ab column=event:timestamp, timestamp=1517159115042, value=2018-01-28T00:25:30.616Z ff91e204-05b0-48aa-a666-7942e3f109ab column=event:timestampMillis, timestamp=1517159115042, value=1517099130616 ff91e204-05b0-48aa-a666-7942e3f109ab column=event:updatedAttributes, timestamp=1517159115042, value={"RouteOnAttribute.Route":"humidity"} ffde140c-3053-4b9d-89c6-14b68025384d column=event:actorHostname, timestamp=1517159114898, value=192.168.1.193 ffde140c-3053-4b9d-89c6-14b68025384d column=event:application, timestamp=1517159114898, value=NiFi Flow ffde140c-3053-4b9d-89c6-14b68025384d column=event:childIds, timestamp=1517159114898, value=[] ffde140c-3053-4b9d-89c6-14b68025384d column=event:componentId, timestamp=1517159114898, value=3a25cda9-0161-1000-813c-631724a10585 ffde140c-3053-4b9d-89c6-14b68025384d column=event:componentName, timestamp=1517159114898, value=RouteOnAttribute ffde140c-3053-4b9d-89c6-14b68025384d column=event:componentType, timestamp=1517159114898, value=RouteOnAttribute ffde140c-3053-4b9d-89c6-14b68025384d column=event:contentURI, timestamp=1517159114898, value=http://192.168.1.193:8080/nifi-api/provenance-events/61701/content/output ffde140c-3053-4b9d-89c6-14b68025384d column=event:durationMillis, timestamp=1517159114898, value=-1 ffde140c-3053-4b9d-89c6-14b68025384d column=event:entityId, timestamp=1517159114898, value=9b017666-7ce9-45c5-9d0a-2f81e56d6fa8 ffde140c-3053-4b9d-89c6-14b68025384d column=event:entitySize, timestamp=1517159114898, value=16 ffde140c-3053-4b9d-89c6-14b68025384d column=event:entityType, timestamp=1517159114898, value=org.apache.nifi.flowfile.FlowFile ffde140c-3053-4b9d-89c6-14b68025384d column=event:eventOrdinal, timestamp=1517159114898, value=61701 ffde140c-3053-4b9d-89c6-14b68025384d column=event:eventType, timestamp=1517159114898, value=ROUTE ffde140c-3053-4b9d-89c6-14b68025384d column=event:lineageStart, timestamp=1517159114898, value=1517084974341 ffde140c-3053-4b9d-89c6-14b68025384d column=event:parentIds, timestamp=1517159114898, value=[] ffde140c-3053-4b9d-89c6-14b68025384d column=event:platform, timestamp=1517159114898, value=nifi ffde140c-3053-4b9d-89c6-14b68025384d column=event:previousAttributes, timestamp=1517159114898, value={"path":"./","filename":"uv.164064-164080.log","s2s.address":"192.168.1.197:55032","s2s.host":"192.168 .1.197","mime.type":"text/plain","uuid":"9b017666-7ce9-45c5-9d0a-2f81e56d6fa8","tailfile.original.path":"/opt/demo/logs/uv.log"} ffde140c-3053-4b9d-89c6-14b68025384d column=event:previousContentURI, timestamp=1517159114898, value=http://192.168.1.193:8080/nifi-api/provenance-events/61701/content/input ffde140c-3053-4b9d-89c6-14b68025384d column=event:previousEntitySize, timestamp=1517159114898, value=16 ffde140c-3053-4b9d-89c6-14b68025384d column=event:processGroupId, timestamp=1517159114898, value=01611005-4e82-1491-ae5d-ca64f59491cb ffde140c-3053-4b9d-89c6-14b68025384d column=event:processGroupName, timestamp=1517159114898, value=Process MiniFi Creator ffde140c-3053-4b9d-89c6-14b68025384d column=event:timestamp, timestamp=1517159114898, value=2018-01-28T00:25:30.607Z ffde140c-3053-4b9d-89c6-14b68025384d column=event:timestampMillis, timestamp=1517159114898, value=1517099130607 ffde140c-3053-4b9d-89c6-14b68025384d column=event:updatedAttributes, timestamp=1517159114898, value={"RouteOnAttribute.Route":"uv"} 1830 row(s) in 11.7680 seconds provenancereporting.xml Learning to Use HBase https://hortonworks.com/hadoop-tutorial/introduction-apache-hbase-concepts-apache-phoenix-new-backup-restore-utility-hbase/ ... View more

I have just started working on updated Apache Tika and Apache OpenNLP processors for Apache 1.5 and while testing found an interesting workflow I would like to share. I am using a few of my processors in this flow: https://github.com/tspannhw/nifi-attributecleaner-processor <- Just updated this https://github.com/tspannhw/nifi-nlp-processor https://github.com/tspannhw/nifi-extracttext-processor <- this needs a major version refresh https://github.com/tspannhw/nifi-corenlp-processor Here is the flow that I was working on. Step 1 - Load Some PDFs Step 2 - Use the built-in Apache Tika Processor to extract metadata from the files Step 3 - Pull Out the Text using my Apache Tika processor Step 4 - Split this into individual lines Step 5 - Extract out the text of the line into an attribute ((^.*$)) into a sentence Step 6 - Run NLP to analyze for names and locations on that sentence Step 7 - Run Stanford CoreNLP sentiment analysis on the sentence Step 8 - I run my attribute cleaner to turn those attributes into AVRO safe names Step 9 - I turn all the attributes into a JSON Flow File Step 10 - I Infer an Avro schema ( I only needed this once, then I'll remove it) Step 11 - I set the name of the Schema to be looked up from the Schema Registry Step 12 - I run QueryRecord to route POSITIVE, NEURAL and NEGATIVE sentiment to different places. Example SQL: SELECT * FROM FLOWFILE WHERE sentiment = 'NEGATIVE' Thanks Apache Calcite! We also convert from JSON to AVRO for sending to Kafka also for easy conversion to Apache ORC for Apache Hive usage. Step 13-14-15 - I send records to Kafka 1.0, Some get merged to store as a file and some get made into Slack messages. Step 16. Done Here is an example of my generated JSON file. Here are some of the attributes after the run. ' You can see the queries in the QueryRecord processor. The results of a run showing a sentence, file meta data and sentiment. ' We are now waiting for new PDFs (and other file types) to arrive in the directory for immediate processing. I have a JSONTreeReader, a Hortonworks Schema Registry and and AvroRecordSetWriter. We set the properties and the schema register for the reader and writer. Obviously we can use other readers and writers as needed for types like CSV. When I am done, since it's Apache NiFi 1.5, I commit my changes for versioning. Bam! tika.xml For the upcoming processor I will be interfacing with: https://wiki.apache.org/tika/TikaOCR https://wiki.apache.org/tika/TikaAndNER https://wiki.apache.org/tika/TikaAndNLTK https://wiki.apache.org/tika/GrobidQuantitiesParser https://wiki.apache.org/tika/TikaAndMITIE https://wiki.apache.org/tika/AgeDetectionParser https://wiki.apache.org/tika/TikaAndVision https://wiki.apache.org/tika/TikaAndVisionDL4J https://wiki.apache.org/tika/ImageCaption Apache Tika has added some really cool updates, so I can't wait to dive in. ... View more

For 2018, some awesome new in-demand features have come to my favorite Swiss Army Knife of IoT and Enterprise development, Apache NiFi. Speaking of knives, for fun say, “Apache NiFi” to Google Assistant. Okay, back to the awesome new release of Apache NiFi. So there are a couple of new Processors that I want to highlight first. I am liking the new CountText processor, useful for counting elements of text documents like words and lines. My example flow is using it and I see some useful metrics gathering there. I also think some of these could be used as file validation checks to feed to machine learning algorithms. My files of type X are usually of this # of lines and words, but not this time. I have come across a couple of use cases on file ingest in the past that could use this. In one example a company was reading personnel files from an sFTP. The first step in validation was checking to see they received the proper number of lines, since one person per line. Another occasion sometimes the client would receive bad files in FTP, they looked fine but the last few records in a file would be missing so they needed to meet a minimum number of characters in the file. In yet another they were counting words for legal documents. Example Run text.line.count 1 No value set text.word.count Another cool processor that I will talk about in greater detail in future articles is the much-requested Spark Processor. The ExecuteSparkInteractive processor with its Livy Controller Service gives you a much better alternative to my hacky REST integration https://community.hortonworks.com/articles/148730/integrating-apache-spark-2x-jobs-with-apache-nifi.html to calling Apache Spark batch and machine learning jobs. There are a number of enhancements, new processors and upgrades I’m excited about, but the main reason I am writing today is because of a new feature that allows for having an agile SDLC with Apache NiFi. This is now enabled by Apache NiFi Registry. It’s as simple as a quick git clone or download and then use Apache Maven to install Apache NiFi Registry and start it. This process will become even easier with future Ambari integration for a CLI-free install. To integrate the Registry with Apache NiFi, you need to add a Registry Client. It’s very simple, to add the default local one, see below. Accessing Apache NiFi Registry By default, it will be running here: http://localhost:18080/nifi-registry/ I did a quick install and did not set any security, with the next HDF release everything will be integrated and simple. Accessing Apache NiFi Flow Registry API As is the case with Apache NiFi, there is a great REST API that comes with the new Apache NiFi Registry. This API is very well documented and easy to follow. This will allow for easy integration with all the popular DevOps automation tools which will please all the DevOps focused teams out there. http://localhost:18080/nifi-registry-api/buckets http://localhost:18080/nifi-registry-api/items http://localhost:18080/nifi-registry-api/tenants/user-groups http://localhost:18080/nifi-registry-api/tenants/users http://localhost:18080/nifi-registry-api/policies http://localhost:18080/nifi-registry-api/access Example Output: {"identity":"anonymous","anonymous":true,"resourcePermissions":{"buckets":{"canRead":true,"canWrite":true,"canDelete":true},"tenants":{"canRead":true,"canWrite":true,"canDelete":true},"policies":{"canRead":true,"canWrite":true,"canDelete":true},"proxy":{"canRead":true,"canWrite":true,"canDelete":true},"anyTopLevelResource":{"canRead":true,"canWrite":true,"canDelete":true}}} I added a few buckets to try out. After you have done that, now you can start using it in Apache NiFi. It could not be easier. Step 1: Create or use an existing Processor Group. Step 2: Right click and pick Version – Start version control. You then pick a Registry (if you have more than one) and a bucket. A bucket is a logical categorization of related flows. I created buckets for Development, Testing and Production. You then add a name, description, and comments for this particular flow and then SAVE. You have just versioned a Process Group. You can now run agile team development with Apache NiFi in your enterprise with familiar version control, team development and isolation. You now have a versioned Processor Group: Now you can edit your flow and see that it has changed. You can now easily commit those changes or revert. To see what changed just pick “Show local changes” As you can see you get a very slick display of what changed to what component. Step 3: Now let’s jump to Apache NiFi Registry and see what happened. The above screenshot shows that my flow “Nifi 1.5 Test” has been stored in bucket “Tim” and has 3 saved versions. An Example Versioned Test Flow Now that your flow is version controlled, others can import that into their workspace (depending on security). You can choose from any of the versions based on your needs. For teams, this part is awesome: You will know if there’s a newer version and you can pick that one if you wish. Or not. You can run many copies of the same flow with different variables and versions. My next article will be around updates to integrating with Apache Spark via Apache Livy. Other Steps Change to Another Version Commit Your Local Changes (or Revert Them) Save Your Flow Version to Any Bucket or Registry You have Permissions To Your Variable Registry is Per Versioned Processor Group This is the second version I am saving. Add some comments. New Sub-Project, Processors, Tasks and Services: MoveHDFS Processor Kafka 1.0 Processors CSVRecordLookupService New Graphite Reporting Task Spark Job Executor with Apache Livy Integration FlattenJSON Processor DeleteMongo Processor TextCount Processor Apache NiFi Registry Resources: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.5.0 https://github.com/apache/nifihttps://nifi.apache.org/download.html https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12341668 Apache NiFi Registry https://nifi.apache.org/docs/nifi-registry-docs/index.html https://github.com/apache/nifi-registry#getting-started https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiRegistry0.1.0 ... View more

Initial Login to ODroid XU4 GPS Command Line Check WiFi Signal Strength Install Ubuntu Extras Above is our device with GPS The ODroid XU4 is a powerful little inexpensive IoT device. It is similar to a Raspberry PI and compatible with many of the libraries. I chose to install ARMBIAN Ubuntu which is a good OS. It is very easy to add GPS libraries, Apache MiniFi, JDK 8 and Python for processing. Apache NiFi processing flow Apache MiniFi Flow on Device Apache NiFi Processing See: https://community.hortonworks.com/content/kbentry/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html Source Code https://github.com/tspannhw/odroidxu4-gps-python-minifi Software sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade armbian-config sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer -y sudo apt-get install oracle-java8-set-default -y blkid swapon /dev/sda1 sudo apt-get install gpsd gpsd-clients sudo dpkg-reconfigure gpsd sudo gpsmon /dev/ttyACM0 sudo apt-get install python-gps cgps -s Resources http://www.hardkernel.com/main/products/prdt_info.php?g_code=G148048570542 https://wiki.odroid.com/odroid-xu4/gettingstart/linux/install_linux https://magazine.odroid.com https://magazine.odroid.com/wp-content/uploads/odroid-xu4-user-manual.pdf#page=6 https://magazine.odroid.com/article/running-yolo-odroid-yolodroid/ http://machinethink.net/blog/object-detection-with-yolo/ https://www.armbian.com/odroid-xu4/ https://docs.armbian.com/ https://docs.armbian.com/User-Guide_Getting-Started/ https://forum.armbian.com/topic/1768-booting-into-command-line-on-opi-pc/ https://docs.armbian.com/User-Guide_Armbian-Config/ https://magazine.odroid.com/wp-content/uploads/ODROID-Magazine-201712.pdf https://magazine.odroid.com/article/running-yolo-odroid-yolodroid/ https://wiki.odroid.com/odroid-xu4/odroid-xu4 https://www.packtpub.com/books/content/working-webcam-and-pi-camera Hardware Armbian Ubuntu 16.04.3 LTS 4.9.61-odroidxu4 Two quad core CPUs and a GPU 2GB LPDDR3 USB 3 Power Supply Powered USB Hub Active Cooling Fan Stratus GPYes USB GPS Device Example Run of GPS Device root@odroidxu4:/opt/demo# ./gps2.sh [{"utc": "2018-01-02T00:19:25.000Z", "epx": "25.829", "epv": "62.56", "diskfree": "54166.8 MB", "altitude": "41.844", "memory": 7.6, "eps": "1.43", "longitude": "-74.529222193", "ts": "2018-01-02 00:19:23", "ept": "0.005", "host": "odroidxu4", "track": "141.59", "mode": "3", "time": "2018-01-02T00:19:25.000Z", "latitude": "40.268098177", "climb": "-0.042", "ipaddress": "192.168.1.202", "cputemp": 53.0, "speed": "0.189"}] Software and Device Setup We need to setup a Swap area to deal with not enough RAM for builds. root@odroidxu4:~# sudo blkid /dev/mmcblk1p1: UUID="7a81b72b-eee2-428f-9706-2aa0bd7e766f" TYPE="ext4" PARTUUID="7cbb190c-01" /dev/zram7: UUID="794ef788-2696-4f56-97da-8c9e179be92d" TYPE="swap" /dev/zram0: UUID="99613ecb-d41b-4dbe-a85c-2134800fcd8e" TYPE="swap" /dev/zram1: UUID="57aaeb7c-b970-456d-a415-3e9151dc5fbf" TYPE="swap" /dev/zram2: UUID="61a42b3a-e605-49e4-844b-467212228878" TYPE="swap" /dev/zram3: UUID="7f7539df-3500-489b-990b-929d09896a42" TYPE="swap" /dev/zram4: UUID="49d3a36a-5150-4c31-8acc-c77c512f8a31" TYPE="swap" /dev/zram5: UUID="9ae98fa0-507d-415a-95d3-e24e9a4a8b2f" TYPE="swap" /dev/zram6: UUID="4644d45f-dd3e-4bab-9507-1e28fe4b9d40" TYPE="swap" /dev/mmcblk1: PTUUID="7cbb190c" PTTYPE="dos" /dev/sda1: UUID="3C09-B728" TYPE="vfat" root@odroidxu4:~# sudo mkswap /dev/sda1 mkswap: /dev/sda1: warning: wiping old vfat signature. Setting up swapspace version 1, size = 114.6 GiB (123010527232 bytes) no label, UUID=24360a18-0939-47b1-b383-27982d8a2007 root@odroidxu4:~# sudo swapon /dev/sda1 root@odroidxu4:~# sudo swapon NAME TYPE SIZE USED PRIO /dev/zram0 partition 124.6M 0B 5 /dev/zram1 partition 124.6M 0B 5 /dev/zram2 partition 124.6M 0B 5 /dev/zram3 partition 124.6M 0B 5 /dev/zram4 partition 124.6M 0B 5 /dev/zram5 partition 124.6M 0B 5 /dev/zram6 partition 124.6M 0B 5 /dev/zram7 partition 124.6M 0B 5 /dev/sda1 partition 114.6G 0B -1 sudo apt-get install gpsd gpsd-clients sudo gpsmon /dev/ttyACM0 sudo cgps -s sudo apt-get install python-gps sudo apt-get install gpsd gpsd-clients sudo dpkg-reconfigure gpsd sudo gpsmon /dev/ttyACM0 sudo pip install psutil sudo pip install --upgrade pip sudo pip install psutil ___ _ _ _ __ ___ _ _ _ / _ \ __| |_ __ ___ (_) __| | \ \/ / | | | || | | | | |/ _` | '__/ _ \| |/ _` | \ /| | | | || |_ | |_| | (_| | | | (_) | | (_| | / \| |_| |__ _| \___/ \__,_|_| \___/|_|\__,_| /_/\_\___/ |_| Welcome to ARMBIAN 5.36 user-built Ubuntu 16.04.3 LTS 4.9.61-odroidxu4 System load: 0.00 0.01 0.00 Up time: 10 min Memory usage: 3 % of 1993MB IP: 192.168.1.202 CPU temp: 58°C Usage of /: 6% of 57G [ General system configuration (beta): armbian-config ] Last login: Mon Jan 1 23:19:21 2018 from 192.168.1.193 pip install --upgrade pip setuptools wheel pip install -upgrade psutil<br> ... View more