Community Articles

TimothySpann · ‎01-02-2018

This is pretty close to a Raspberry Pi Zero. This inexpensive box is another useful IoT device for ingesting some data very inexpensively.

Machine Browsing

root@NanoPi-Duo:/home/pi# cpu_freq
INFO: HARDWARE=sun8i
CPU0 online=1 temp=48092 governor=ondemand cur_freq=312000
CPU1 online=1 temp=48092 governor=ondemand cur_freq=312000
CPU2 online=1 temp=48092 governor=ondemand cur_freq=312000
CPU3 online=1 temp=48092 governor=ondemand cur_freq=312000

Source Code for Python and Shell Script

https://github.com/tspannhw/nifi-nanopi-duo

Hardware

The FA-CAM202 is a 200M USB camera.

512MB RAM

Ubuntu 16.04.3 LTS 4.11.2

Similar SBC to Raspberry PI

ARM

Software Setup

sudo apt-get install fswebcam -y

sudo apt-get install libv4l-dev -y

sudo apt-get install python-opencv -y

sudo npi-config

sudo apt-get update

sudo apt-get install libcv-dev libopencv-dev -y

pip install psutil

pip2 install psutil

curl http://192.168.1.193:8080/nifi-api/site-to-site/ -v

inferred.avro.schema

{ "type" : "record", "name" : "NANO", "fields" : [ { "name" : "diskfree", "type" : "string", "doc" : "Type inferred from '\"23329.1 MB\"'" }, { "name" : "cputemp", "type" : "double", "doc" : "Type inferred from '55.0'" }, { "name" : "host", "type" : "string", "doc" : "Type inferred from '\"NanoPi-Duo\"'" }, { "name" : "endtime", "type" : "string", "doc" : "Type inferred from '\"2018-01-04 20:23:56\"'" }, { "name" : "ipaddress", "type" : "string", "doc" : "Type inferred from '\"192.168.1.191\"'" }, { "name" : "h", "type" : "int", "doc" : "Type inferred from '342'" }, { "name" : "ts", "type" : "string", "doc" : "Type inferred from '\"2018-01-04 20:23:43\"'" }, { "name" : "filename", "type" : "string", "doc" : "Type inferred from '\"/opt/demo/images/2018-01-04_2023.jpg.faces.jpg\"'" }, { "name" : "w", "type" : "int", "doc" : "Type inferred from '342'" }, { "name" : "memory", "type" : "double", "doc" : "Type inferred from '60.1'" }, { "name" : "y", "type" : "int", "doc" : "Type inferred from '264'" }, { "name" : "x", "type" : "int", "doc" : "Type inferred from '877'" } ] }

hive.ddl

CREATE EXTERNAL TABLE IF NOT EXISTS nanopi (diskfree STRING, cputemp DOUBLE, host STRING, endtime STRING, ipaddress STRING, h INT, ts STRING, filename STRING, w INT, memory DOUBLE, y INT, x INT) STORED AS ORC LOCATION '/nano'

Apache NiFi

The flow in Apache NiFi is pretty simple. We receive the flowfiles from the remote Apache MiniFi box.

1. RouteOnAttribute: Send images to the file system, continue processing the JSON

2. AttributeCleanerProcessor: Clean up the attributes Not really needed for this dataset.

3. UpdateAttribute: Set the schema name to reference the registry

4. SplitJSON: Split JSON into one array per flowfile

5. ConvertRecord: Convert JSON Tree to AVRO with embedded schema

6. ConvertAvroToORC: Build an Apache ORC file (we could add a step before for MergeContent)

7. PutHDFS: Store in Hadoop File System forever

Apache MiniFi

We have a simple flow in Apache MiniFi.

1. ExecuteProcess: Run a shell script to grab a timestamp filenamed image from the USB webcam.

Then call a Python script that does OpenCV Face Detection and adds some local variables to a JSON array with facial squares.

2. GetFile: retrieve all the images from the box and send them to Apache NiFi.

Example Data

[{"diskfree": "23445.5 MB", "cputemp": 56.7, "host": "NanoPi-Duo", "endtime": "2018-01-04 17:40:30", "ipaddress": "192.168.1.191", "h": 55, "ts": "2018-01-04 17:40:20", "filename": "/opt/demo/images/2018-01-04_1740.jpg.faces.jpg", "w": 55, "memory": 22.6, "y": 471, "x": 270}, {"diskfree": "23445.5 MB", "cputemp": 56.7, "host": "NanoPi-Duo", "endtime": "2018-01-04 17:40:30", "ipaddress": "192.168.1.191", "h": 67, "ts": "2018-01-04 17:40:20", "filename": "/opt/demo/images/2018-01-04_1740.jpg.faces.jpg", "w": 67, "memory": 22.6, "y": 625, "x": 464}]

Resources

Face

Cloudera Community

Community Articles

IOT: Ingesting Camera Data from NanoPi DUO Devices with Apache MiniFi and Python

Apache MiNiFi