For this edge use case we are using NVidia's TensorRT as well as Apache MXNet. From TensorRT I am using imageNet for image recognition and detectNet for object localization. For Apache MXNet, I am using their image classifier. So we have multiple deep learning frameworks run on the same capture from an attached USB webcam. For this example I am using a Logitech HD1080, while the Jetson TX1 supports 6+ concurrent high end cameras for those with high end use cases. They also have a more powerful Jetson TX2 for more intense use cases as it has more RAM and a better GPU. Quick Hardware Breakdown NVIDIA Maxwell™ GPU with 256 NVIDIA® CUDA® Cores 4 GB LPDDR4 Memory Python Script # 2017 load pictures and analyze # https://github.com/tspannhw/mxnet_rpi/blob/master/analyze.py import time import sys import datetime import subprocess import sys import urllib2 import os import datetime import traceback import math import random, string import base64 import json from time import gmtime, strftime import mxnet as mx import inception_predict import numpy as np import cv2 import math import random, string import time from time import gmtime, strftime start = time.time() cap = cv2.VideoCapture(0) packet_size=3000 def randomword(length): return ''.join(random.choice(string.lowercase) for i in range(length)) #while True: # Create unique image name uniqueid = 'mxnet_uuid_{0}_{1}'.format(randomword(3),strftime("%Y%m%d%H%M%S",gmtime())) ret, frame = cap.read() imgdir = 'images/' filename = 'tx1_image_{0}_{1}.jpg'.format(randomword(3),strftime("%Y%m%d%H%M%S",gmtime())) cv2.imwrite(imgdir + filename, frame) # Run inception prediction on image try: topn = inception_predict.predict_from_local_file(imgdir + filename, N=5) except: errorcondition = "true" # CPU Temp f = open("/sys/devices/virtual/thermal/thermal_zone1/temp","r") cputemp = str( f.readline() ) cputemp = cputemp.replace('\n','') cputemp = cputemp.strip() cputemp = str(round(float(cputemp)) / 1000) cputempf = str(round(9.0/5.0 * float(cputemp) + 32)) f.close() # GPU Temp f = open("/sys/devices/virtual/thermal/thermal_zone2/temp","r") gputemp = str( f.readline() ) gputemp = gputemp.replace('\n','') gputemp = gputemp.strip() gputemp = str(round(float(gputemp)) / 1000) gputempf = str(round(9.0/5.0 * float(gputemp) + 32)) f.close() # Face Detect p = os.popen('/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/facedetect.sh ' + filename).read() face = p.replace('\n','|') face = face.strip() # NVidia Image Net Classify p2 = os.popen('/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/runclassify.sh ' + filename).read() imagenet = p2.replace('\n','|') imagenet = imagenet.strip() # 5 MXNET Analysis top1 = str(topn[0][1]) top1pct = str(round(topn[0][0],3) * 100) top2 = str(topn[1][1]) top2pct = str(round(topn[1][0],3) * 100) top3 = str(topn[2][1]) top3pct = str(round(topn[2][0],3) * 100) top4 = str(topn[3][1]) top4pct = str(round(topn[3][0],3) * 100) top5 = str(topn[4][1]) top5pct = str(round(topn[4][0],3) * 100) end = time.time() # face[-4096:] row = { 'uuid': uniqueid, 'top1pct': top1pct, 'top1': top1, 'top2pct': top2pct, 'top2': top2,'top3pct': top3pct, 'top3': top3,'top4pct': top4pct,'top4': top4, 'top5pct': top5pct,'top5': top5, 'cputemp': cputemp, 'gputemp': gputemp, 'imagefilename': filename, 'gputempf': gputempf, 'cputempf': cputempf, 'runtime': str(round(end - start)), 'facedetect': face, 'imagenet': imagenet } json_string = json.dumps(row) print (json_string ) Setup Jetson TX1 for Deep Learning and Computer Vision sudo apt-get update -y sudo apt-get -y install git build-essential libatlas-base-dev libopencv-dev graphviz python-pip sudo pip install pip --upgrade sudo pip install setuptools numpy --upgrade Apache Hive DDL CREATE EXTERNAL TABLE IF NOT EXISTS jetsonscan (top3pct STRING, uuid STRING, top1pct STRING, top5 STRING, top4 STRING, top3 STRING, top2 STRING, top1 STRING, top4pct STRING, facedetect STRING, gputempf STRING, gputemp STRING, top5pct STRING, top2pct STRING, cputemp STRING, imagenet STRING, runtime STRING, imagefilename STRING, cputempf STRING) STORED AS ORC LOCATION '/jetsonscan' Build Apache MiniFi Configuration minifi-toolkit-0.2.0/bin/config.sh transform $1 config.yml scp config.yml nvidia@192.168.1.190:/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/minifi-0.2.0/conf/ Example Output JSON { "top3pct" : "6.1", "uuid" : "mxnet_uuid_pgo_20171110193628", "top1pct" : "8.3", "top5" : "n03110669 cornet, horn, trumpet, trump", "top4" : "n03481172 hammer", "top3" : "n02787622 banjo", "top2" : "n02791270 barbershop", "top1" : "n04487394 trombone", "top4pct" : "4.4", "facedetect" : "networks/facenet-120/snapshot_iter_24000.caffemodel initialized.|[cuda] cudaAllocMapped 16 bytes, CPU 0x1013a0000 GPU 0x1013a0000|maximum bounding boxes: 3136|[cuda] cudaAllocMapped 50176 bytes, CPU 0x1012a6200 GPU 0x1012a6200|[cuda] cudaAllocMapped 12544 bytes, CPU 0x1011a1a00 GPU 0x1011a1a00|failed to load image /media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg|failed to load image '/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg'|", "gputempf" : "68.0", "gputemp" : "20.0", "top5pct" : "3.2", "top2pct" : "6.4", "cputemp" : "21.5", "imagenet" : "imagenet-console| args (3): 0 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/imagenet-console] 1 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg] 2 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/cfout-tx1_image_xmv_20171110193629.jpg] |||imageNet -- loading classification network model from:| -- prototxt networks/googlenet.prototxt| -- model networks/bvlc_googlenet.caffemodel| -- class_labels networks/ilsvrc12_synset_words.txt| -- input_blob 'data'| -- output_blob 'prob'| -- batch_size 2||[GIE] attempting to open cache file networks/bvlc_googlenet.caffemodel.2.tensorcache|[GIE] loading network profile from cache... networks/bvlc_googlenet.caffemodel.2.tensorcache|[GIE] platform has FP16 support.|[GIE] networks/bvlc_googlenet.caffemodel loaded|[GIE] CUDA engine context initialized with 2 bindings|[GIE] networks/bvlc_googlenet.caffemodel input binding index: 0|[GIE] networks/bvlc_googlenet.caffemodel input dims (b=2 c=3 h=224 w=224) size=1204224|[cuda] cudaAllocMapped 1204224 bytes, CPU 0x100ce0000 GPU 0x100ce0000|[GIE] networks/bvlc_googlenet.caffemodel output 0 prob binding index: 1|[GIE] networks/bvlc_googlenet.caffemodel output 0 prob dims (b=2 c=1000 h=1 w=1) size=8000|[cuda] cudaAllocMapped 8000 bytes, CPU 0x100e20000 GPU 0x100e20000|networks/bvlc_googlenet.caffemodel initialized.|[GIE] networks/bvlc_googlenet.caffemodel loaded|imageNet -- loaded 1000 class info entries|networks/bvlc_googlenet.caffemodel initialized.|failed to load image /media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg|failed to load image '/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/images/tx1_image_xmv_20171110193629.jpg'|", "runtime" : "8.0", "imagefilename" : "tx1_image_xmv_20171110193629.jpg", "cputempf" : "71.0" } Schema (Put this in Hortonworks Schema Registry) - MXRECORD { "type" : "record", "name" : "MXRECORD", "fields" : [ { "name" : "top3pct", "type" : "string", "doc" : "Type inferred from '\"5.0\"'" }, { "name" : "uuid", "type" : "string", "doc" : "Type inferred from '\"mxnet_uuid_ltu_20171110193847\"'" }, { "name" : "top1pct", "type" : "string", "doc" : "Type inferred from '\"5.4\"'" }, { "name" : "top5", "type" : "string", "doc" : "Type inferred from '\"n03970156 plunger, plumber's helper\"'" }, { "name" : "top4", "type" : "string", "doc" : "Type inferred from '\"n07615774 ice lolly, lolly, lollipop, popsicle\"'" }, { "name" : "top3", "type" : "string", "doc" : "Type inferred from '\"n04270147 spatula\"'" }, { "name" : "top2", "type" : "string", "doc" : "Type inferred from '\"n03110669 cornet, horn, trumpet, trump\"'" }, { "name" : "top1", "type" : "string", "doc" : "Type inferred from '\"n04487394 trombone\"'" }, { "name" : "top4pct", "type" : "string", "doc" : "Type inferred from '\"4.5\"'" }, { "name" : "facedetect", "type" : "string" }, { "name" : "gputempf", "type" : "string", "doc" : "Type inferred from '\"68.0\"'" }, { "name" : "gputemp", "type" : "string", "doc" : "Type inferred from '\"20.0\"'" }, { "name" : "top5pct", "type" : "string", "doc" : "Type inferred from '\"4.4\"'" }, { "name" : "top2pct", "type" : "string", "doc" : "Type inferred from '\"5.3\"'" }, { "name" : "cputemp", "type" : "string", "doc" : "Type inferred from '\"23.0\"'" }, { "name" : "imagenet", "type" : "string" }, { "name" : "runtime", "type" : "string", "doc" : "Type inferred from '\"8.0\"'" }, { "name" : "imagefilename", "type" : "string", "doc" : "Type inferred from '\"tx1_image_okg_20171110193848.jpg\"'" }, { "name" : "cputempf", "type" : "string", "doc" : "Type inferred from '\"73.0\"'" } ] } Example Apache MiniFi Logs 2017-11-10 15:13:53,061 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 51004 2017-11-10 15:13:53,084 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.lucene.SimpleIndexManager Index Writer for provenance_repository/index-1503524885000 has been returned to Index Manager and is no longer in use. Closing Index Writer 2017-11-10 15:13:53,086 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (6 records) into single Provenance Log File provenance_repository/50998.prov in 28 milliseconds 2017-11-10 15:13:53,087 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 70 records. In the past 5 minutes, 29 events have been written to the Provenance Repository, totaling 18.54 KB 2017-11-10 15:14:08,531 INFO [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@60bcd09e Successfully refreshed Peer Status; remote instance consists of 1 peers 2017-11-10 15:14:38,658 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] [15:14:38] src/nnvm/legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade... 2017-11-10 15:14:38,665 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] [15:14:38] src/nnvm/legacy_json_util.cc:198: Symbol successfully upgraded! 2017-11-10 15:14:38,716 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] /media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/mxnet/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label']) 2017-11-10 15:14:38,717 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] warnings.warn(msg) 2017-11-10 15:14:38,965 WARN [ExecuteProcess c216f845-1839-3f3c-0000-000000000000 Task] o.a.n.processors.standard.ExecuteProcess ExecuteProcess[id=c216f845-1839-3f3c-0000-000000000000] HIGHGUI ERROR: V4L/V4L2: VIDIOC_S_CROP Resources https://github.com/tspannhw/nvidiajetsontx1-mxnet https://developer.nvidia.com/embedded/twodaystoademo https://github.com/dusty-nv/jetson-inference https://developer.nvidia.com/tensorrt https://developer.nvidia.com/embedded/buy/jetson-tx1-devkit https://github.com/tspannhw/nvidiajetsontx1-mxnet Flow Files storejetsontx1.xml jetsontx1mx-10nov2017.xml ... View more

If you have not attended a DataWorksSummit, I highly recommend it. It is an amazing event held at three locations a year and is a great community experience. The content is deep and highly technical and you will learn about the current state of the art and what is coming next. It's not just Big Data, but AI, Streaming, Microservices, Containers, Cloud and many other topics that startups and enterprises alike need to know. My topic was a simple talk on using Apache NiFi to ingest and transform various data types. There is a small group forming around my quickly released Inception V3 TensorFlow Apache NiFi Processor, I encourage you to try it and provide feedback, pull requests, bug reports, documentation, unit tests, examples and more. The Java API for TensorFlow is new so this is really basic. Thanks to @Simon Elliston Ball for a major cleanup on it. https://github.com/tspannhw/nifi-tensorflow-processor What do we want to do? MiniFi ingests camera images and sensor data Run TensorFlow Inception v3 to recognize objects in image NiFi stores images, metadata and enriched data in Hadoop NiFi ingests social data and feeds NiFi analyzes sentiment of textual data •TensorFlow (C++, Python, Java) via ExecuteStreamCommand • •TensorFlow NiFi Java Custom Processor • •TensorFlow Running on Edge Nodes (MiniFi) • • • •TensorFlow Mobile (iOS, Android, RPi) • •TensorFlow on Spark (Yahoo) via Livy, S2S, Kafka • •TensorFlow Running in Containers in YARN 3.0 on Hadoop • (NiFI 1.4) gRPC Call to TensorFlow Serving python classify_image.py --image_file/dir/solarroofpanel.jpg<br>solar dish, solar collector, solar furnace (score = 0.98316)<br>window screen (score = 0.00196)<br>manhole cover (score = 0.00070)<br>radiator (score = 0.00041)<br>doormat, welcome mat (score = 0.00041) Python Uses pip install -U textblob python -m textblob.download_corpora pip install -U spacy python -m spacy.en.download all pip install -U nltk pip install -U numpy run.sh python sentiment.py "$@” sentiment.py sentiment.pyfrom nltk.sentiment.vader import SentimentIntensityAnalyzer import sys sid = SentimentIntensityAnalyzer() ss = sid.polarity_scores(sys.argv[1]) print('Compound {0} Negative {1} Neutral {2} Positive {3} '.format( ss['compound'],ss['neg'],ss['neu'],ss['pos'])) These are some good Python libraries to be using. I recommend using Python 3.X unless you are stuck with 2.6/2.7. I have also created two processors for working with text/NLP, these are listed below for Apache OpenNLP and Stanford CoreNLP. Please comment in HCC (here), check out github and do pull requests (https://github.com/tspannhw) and come to a meetup (https://www.meetup.com/futureofdata-princeton/). References: https://github.com/tspannhw/dws2017sydney https://dataworkssummit.com/sydney-2017/sessions/real-time-ingesting-and-transforming-sensor-data-and-social-data-with-nifi-and-tensorflow/ https://www.slideshare.net/Hadoop_Summit/realtime-ingesting-and-transforming-sensor-data-and-social-data-with-nifi-and-tensorflow https://hortonworks.com/blog/7-sessions-dataworks-summit-sydney-see/ https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25.html http://www.nltk.org/install.html https://github.com/tspannhw/nifi-nlp- processor https://community.hortonworks.com/articles/80418/open-nlp-example-apache-nifi-processor.html https://community.hortonworks.com/articles/81270/adding-stanford-corenlp-to-big-data-pipelines-apac-1.html ... View more