MiniFi For Image Capture and Ingestion from Raspberry PI
Our Apache NiFi 1.2 instance is waiting for data to be pushed from MiniFi instance.
We export this to an XML template to convert to YAML.
minifi-toolkit-1.0.2.1.4.0-5/bin/config.sh transform MinifiImage.xml config.yml
Then I scp over the YAML file to my MiniFi RPI 3.
A clean running MiniFi instance. Not there will be some gzipped logs showing up in there.
Grab all the images that are available (make sure they are complete and at least 30 seconds old and bigger than 4096 bytes).
Schedule when you want things to run. Our use case is one picture a minute.
Make sure you map your remote process group to the correct Input Port!
MiniFi Setup
Make sure you have Oracle's JDK 8 installed and in the path.
export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/
bin/minifi.sh start
Check the Logs
tail -f log/minifi-app.log
Check the Status
bin/minifi.sh flowStatus controllerservices:health,bulletins
Test your Camera
raspistill -o cam4.jpg
Python Code
#!/usr/bin/python
# heavily remixed existing code
import os
import datetime
import traceback
import math
import random, string
import base64
import picamera
from time import sleep
from time import gmtime, strftime
def randomword(length):
return ''.join(random.choice(string.lowercase) for i in range(length))
# Create unique image name
img_name = '/opt/demo/images/pi_image_{0}_{1}.jpg'.format(randomword(3),strftime("%Y%m%d%H%M%S",gmtime()))
# Capture Image from Pi Camera
try:
camera = picamera.PiCamera()
camera.annotate_text = " Stored with Apache NiFi "
camera.capture(img_name, resize=(800,600))
pass
finally:
camera.close()
Create an image using the Raspberry Pi official camera (I am using Noir). Add image text to each image. I place that script in a shell script and call it from MiniFi. I could use an execute script, I'll do that in the next one.
/opt/demo/runimage.sh
Using MiniFi to send over my images is easier (no weird conversions of binary data), safer, includes more provenance and is preferred over sending with MQTT. If my machine is too tiny (Onion Omega) or locked down, I would use MQTT.
References
- https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.4/bk_minifi-quick-start/content/ch_minifi-qui...
- https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.4/bk_release-notes/content/index.html
- https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.2/bk_dataflow-user-guide/content/configure-si...
- https://community.hortonworks.com/articles/16461/nifi-understanding-how-to-use-process-groups-and-r....
- https://www.ncdc.noaa.gov/cdo-web/datasets
- https://github.com/tspannhw/rpi-picamera-mqtt-nifi
- https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html
- http://www.pyimagesearch.com/2015/03/30/accessing-the-raspberry-pi-camera-with-opencv-and-python/
- https://community.hortonworks.com/content/kbentry/77988/ingest-remote-camera-images-from-raspberry-p...
- https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt....
