For this edge use case we are using NVidia's TensorRT as well as Apache MXNet. From TensorRT I am using imageNet for image recognition and detectNet for object localization.
For Apache MXNet, I am using their image classifier. So we have multiple deep learning frameworks run on the same capture from an attached USB webcam. For this example I am using a Logitech HD1080, while the Jetson TX1 supports 6+ concurrent high end cameras for those with high end use cases. They also have a more powerful Jetson TX2 for more intense use cases as it has more RAM and a better GPU.
Quick Hardware Breakdown
- NVIDIA Maxwell™ GPU with 256 NVIDIA® CUDA® Cores
- 4 GB LPDDR4 Memory
Python Script
# 2017 load pictures and analyze # https://github.com/tspannhw/mxnet_rpi/blob/master/analyze.py import time import sys import datetime import subprocess import sys import urllib2 import os import datetime import traceback import math import random, string import base64 import json from time import gmtime, strftime import mxnet as mx import inception_predict import numpy as np import cv2 import math import random, string import time from time import gmtime, strftime start = time.time() cap = cv2.VideoCapture(0) packet_size=3000 def randomword(length): return ''.join(random.choice(string.lowercase) for i in range(length)) #while True: # Create unique image name uniqueid = 'mxnet_uuid_{0}_{1}'.format(randomword(3),strftime("%Y%m%d%H%M%S",gmtime())) ret, frame = cap.read() imgdir = 'images/' filename = 'tx1_image_{0}_{1}.jpg'.format(randomword(3),strftime("%Y%m%d%H%M%S",gmtime())) cv2.imwrite(imgdir + filename, frame) # Run inception prediction on image try: topn = inception_predict.predict_from_local_file(imgdir + filename, N=5) except: errorcondition = "true" # CPU Temp f = open("/sys/devices/virtual/thermal/thermal_zone1/temp","r") cputemp = str( f.readline() ) cputemp = cputemp.replace('\n','') cputemp = cputemp.strip() cputemp = str(round(float(cputemp)) / 1000) cputempf = str(round(9.0/5.0 * float(cputemp) + 32)) f.close() # GPU Temp f = open("/sys/devices/virtual/thermal/thermal_zone2/temp","r") gputemp = str( f.readline() ) gputemp = gputemp.replace('\n','') gputemp = gputemp.strip() gputemp = str(round(float(gputemp)) / 1000) gputempf = str(round(9.0/5.0 * float(gputemp) + 32)) f.close() # Face Detect p = os.popen('/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/facedetect.sh ' + filename).read() face = p.replace('\n','|') face = face.strip() # NVidia Image Net Classify p2 = os.popen('/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/runclassify.sh ' + filename).read() imagenet = p2.replace('\n','|') imagenet = imagenet.strip() # 5 MXNET Analysis top1 = str(topn[0][1]) top1pct = str(round(topn[0][0],3) * 100) top2 = str(topn[1][1]) top2pct = str(round(topn[1][0],3) * 100) top3 = str(topn[2][1]) top3pct = str(round(topn[2][0],3) * 100) top4 = str(topn[3][1]) top4pct = str(round(topn[3][0],3) * 100) top5 = str(topn[4][1]) top5pct = str(round(topn[4][0],3) * 100) end = time.time() # face[-4096:] row = { 'uuid': uniqueid, 'top1pct': top1pct, 'top1': top1, 'top2pct': top2pct, 'top2': top2,'top3pct': top3pct, 'top3': top3,'top4pct': top4pct,'top4': top4, 'top5pct': top5pct,'top5': top5, 'cputemp': cputemp, 'gputemp': gputemp, 'imagefilename': filename, 'gputempf': gputempf, 'cputempf': cputempf, 'runtime': str(round(end - start)), 'facedetect': face, 'imagenet': imagenet } json_string = json.dumps(row) print (json_string )
Setup Jetson TX1 for Deep Learning and Computer Vision
sudo apt-get update -y sudo apt-get -y install git build-essential libatlas-base-dev libopencv-dev graphviz python-pip sudo pip install pip --upgrade sudo pip install setuptools numpy --upgrade
Apache Hive DDL
CREATE EXTERNAL TABLE IF NOT EXISTS jetsonscan (top3pct STRING, uuid STRING, top1pct STRING, top5 STRING, top4 STRING, top3 STRING, top2 STRING, top1 STRING, top4pct STRING, facedetect STRING, gputempf STRING, gputemp STRING, top5pct STRING, top2pct STRING, cputemp STRING, imagenet STRING, runtime STRING, imagefilename STRING, cputempf STRING) STORED AS ORC LOCATION '/jetsonscan'
Build Apache MiniFi Configuration
minifi-toolkit-0.2.0/bin/config.sh transform $1 config.yml scp config.yml nvidia@192.168.1.190:/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/minifi-0.2.0/conf/
Example Output JSON
{
"top3pct" : "6.1",
"uuid" : "mxnet_uuid_pgo_20171110193628",
"top1pct" : "8.3",
"top5" : "n03110669 cornet, horn, trumpet, trump",
"top4" : "n03481172 hammer",
"top3" : "n02787622 banjo",
"top2" : "n02791270 barbershop",
"top1" : "n04487394 trombone",
"top4pct" : "4.4",
"facedetect" : "networks/facenet-120/snapshot_iter_24000.caffemodel initialized.|[cuda] cudaAllocMapped 16 bytes, CPU 0x1013a0000 GPU 0x1013a0000|maximum bounding boxes: 3136|[cuda] cudaAllocMapped 50176 bytes, CPU 0x1012a6200 GPU 0x1012a6200|[cuda] cudaAllocMapped 12544 bytes, CPU 0x1011a1a00 GPU 0x1011a1a00|failed to load image /media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg|failed to load image '/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg'|",
"gputempf" : "68.0",
"gputemp" : "20.0",
"top5pct" : "3.2",
"top2pct" : "6.4",
"cputemp" : "21.5",
"imagenet" : "imagenet-console| args (3): 0 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/imagenet-console] 1 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg] 2 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/cfout-tx1_image_xmv_20171110193629.jpg] |||imageNet -- loading classification network model from:| -- prototxt networks/googlenet.prototxt| -- model networks/bvlc_googlenet.caffemodel| -- class_labels networks/ilsvrc12_synset_words.txt| -- input_blob 'data'| -- output_blob 'prob'| -- batch_size 2||[GIE] attempting to open cache file networks/bvlc_googlenet.caffemodel.2.tensorcache|[GIE] loading network profile from cache... networks/bvlc_googlenet.caffemodel.2.tensorcache|[GIE] platform has FP16 support.|[GIE] networks/bvlc_googlenet.caffemodel loaded|[GIE] CUDA engine context initialized with 2 bindings|[GIE] networks/bvlc_googlenet.caffemodel input binding index: 0|[GIE] networks/bvlc_googlenet.caffemodel input dims (b=2 c=3 h=224 w=224) size=1204224|[cuda] cudaAllocMapped 1204224 bytes, CPU 0x100ce0000 GPU 0x100ce0000|[GIE] networks/bvlc_googlenet.caffemodel output 0 prob binding index: 1|[GIE] networks/bvlc_googlenet.caffemodel output 0 prob dims (b=2 c=1000 h=1 w=1) size=8000|[cuda] cudaAllocMapped 8000 bytes, CPU 0x100e20000 GPU 0x100e20000|networks/bvlc_googlenet.caffemodel initialized.|[GIE] networks/bvlc_googlenet.caffemodel loaded|imageNet -- loaded 1000 class info entries|networks/bvlc_googlenet.caffemodel initialized.|failed to load image /media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg|failed to load image '/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg'|",
"runtime" : "8.0",
"imagefilename" : "tx1_image_xmv_20171110193629.jpg",
"cputempf" : "71.0"
}
Schema (Put this in Hortonworks Schema Registry) - MXRECORD
{ "type" : "record", "name" : "MXRECORD", "fields" :
[ { "name" : "top3pct", "type" : "string", "doc" : "Type inferred from '\"5.0\"'" },
{ "name" : "uuid", "type" : "string", "doc" : "Type inferred from '\"mxnet_uuid_ltu_20171110193847\"'" },
{ "name" : "top1pct", "type" : "string", "doc" : "Type inferred from '\"5.4\"'" },
{ "name" : "top5", "type" : "string", "doc" : "Type inferred from '\"n03970156 plunger, plumber's helper\"'" },
{ "name" : "top4", "type" : "string", "doc" : "Type inferred from '\"n07615774 ice lolly, lolly, lollipop, popsicle\"'" },
{ "name" : "top3", "type" : "string", "doc" : "Type inferred from '\"n04270147 spatula\"'" },
{ "name" : "top2", "type" : "string", "doc" : "Type inferred from '\"n03110669 cornet, horn, trumpet, trump\"'" },
{ "name" : "top1", "type" : "string", "doc" : "Type inferred from '\"n04487394 trombone\"'" },
{ "name" : "top4pct", "type" : "string", "doc" : "Type inferred from '\"4.5\"'" },
{ "name" : "facedetect", "type" : "string" },
{ "name" : "gputempf", "type" : "string", "doc" : "Type inferred from '\"68.0\"'" },
{ "name" : "gputemp", "type" : "string", "doc" : "Type inferred from '\"20.0\"'" },
{ "name" : "top5pct", "type" : "string", "doc" : "Type inferred from '\"4.4\"'" },
{ "name" : "top2pct", "type" : "string", "doc" : "Type inferred from '\"5.3\"'" },
{ "name" : "cputemp", "type" : "string", "doc" : "Type inferred from '\"23.0\"'" },
{ "name" : "imagenet", "type" : "string" },
{ "name" : "runtime", "type" : "string", "doc" : "Type inferred from '\"8.0\"'" },
{ "name" : "imagefilename", "type" : "string", "doc" : "Type inferred from '\"tx1_image_okg_20171110193848.jpg\"'" },
{ "name" : "cputempf", "type" : "string", "doc" : "Type inferred from '\"73.0\"'" }
]
}
Example Apache MiniFi Logs
2017-11-10 15:13:53,061 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 51004 2017-11-10 15:13:53,084 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.lucene.SimpleIndexManager Index Writer for provenance_repository/index-1503524885000 has been returned to Index Manager and is no longer in use. Closing Index Writer 2017-11-10 15:13:53,086 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (6 records) into single Provenance Log File provenance_repository/50998.prov in 28 milliseconds 2017-11-10 15:13:53,087 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 70 records. In the past 5 minutes, 29 events have been written to the Provenance Repository, totaling 18.54 KB 2017-11-10 15:14:08,531 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@60bcd09e Successfully refreshed Peer Status; remote instance consists of 1 peers 2017-11-10 15:14:38,658 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] [15:14:38] src/nnvm/legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade... 2017-11-10 15:14:38,665 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] [15:14:38] src/nnvm/legacy_json_util.cc:198: Symbol successfully upgraded! 2017-11-10 15:14:38,716 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] /media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/mxnet/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) 2017-11-10 15:14:38,717 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] warnings.warn(msg) 2017-11-10 15:14:38,965 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] HIGHGUI ERROR: V4L/V4L2: VIDIOC_S_CROP
Resources
- https://github.com/tspannhw/nvidiajetsontx1-mxnet
- https://developer.nvidia.com/embedded/twodaystoademo
- https://github.com/dusty-nv/jetson-inference
- https://developer.nvidia.com/tensorrt
- https://developer.nvidia.com/embedded/buy/jetson-tx1-devkit
- https://github.com/tspannhw/nvidiajetsontx1-mxnet
Flow Files
