2019-01-10 10:12:56,417 ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=d2db37b1-0c1a-10cc-7112-fadb5aef597a] Unable to query StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to Field imu_vg_z cannot be null: org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to Field imu_vg_z cannot be null org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to Field imu_vg_z cannot be null at org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(FlowFileEnumerator.java:65) at Baz$1$1.moveNext(Unknown Source) at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:676) at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:96) at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33) at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90) at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:184) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:566) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.nifi.processors.standard.QueryRecord.query(QueryRecord.java:527) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:303) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.nifi.serialization.SchemaValidationException: Field imu_vg_z cannot be null at org.apache.nifi.serialization.record.MapRecord.checkTypes(MapRecord.java:81) at org.apache.nifi.serialization.record.MapRecord.<init>(MapRecord.java:52) at org.apache.nifi.csv.JacksonCSVRecordReader.nextRecord(JacksonCSVRecordReader.java:168) at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50) at org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(FlowFileEnumerator.java:62) ... 26 common frames omitted ... View more

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice In Part 1, we will setup our drone, our communication environment, capture the data and do initial analysis. We will eventually grab live video stream for object detection, real-time flight control and real-time data ingest of photos, videos and sensor readings. We will have Apache NiFi react to live situations facing the drone and have it issue flight commands via UDP. In this initial section, we will control the drone with Python which can be triggered by NiFi. Apache NiFi will ingest log data that is stored as CSV files on a NiFi node connected to the drone's WiFi. This will eventually move to a dedicated embedded device running MiniFi. This is a small personal drone with less than 13 minutes of flight time per battery. This is not a commercial drone, but gives you an idea of the what you can do with drones. Drone Live Communications for Sensor Readings and Drone Control You must connect to the drone's WiFi, which will be Tello(Something). Tello IP: 192.168.10.1 UDP PORT:8889 Receive Tello Video Stream Tello IP: 192.168.10.1 UDP Server: 0.0.0.0 UDP PORT:11111 Example Install: pip3.6 install tellopy git clone https://github.com/hanyazou/TelloPy pip3.6 install av pip3.6 install opencv-python pip3.6 install image python3.6 -m tellopy.examples.video_effect Example Run Video: https://www.youtube.com/watch?v=mYbStkcnhsk&t=0s&list=PL-7XqvSmQqfTSihuoIP_ZAnN7mFIHkZ_e&index=18 Example Flight Log: Tello-flight log.pdf. Let's build a quick ingest with Apache NiFi 1.8. Our first step we use a local Apache NiFi to read the CSV from the Drone rune locally. We read the CSVs from the Tello logging directory, add a schema definition and query it. We have a controller for CSV processing. We are using the posted schema and the Jackson CSV processor. We want to ignore the header as it has invalid characters. We use a QueryRecord to find if the position in Z has changed. SELECT * FROM FLOWFILE WHERE mvo_pos_z is not null AND CAST(mvo_pos_z as FLOAT) <> 0.0 We also convert from CSV to Apache AVRO format for further processing. Valid records are sent over HTTP(S) Site-to-Site to a cloud hosted Apache NiFi cluster for further processing to save to an HBase table. As you can see it's trival to store these records in HBase. For HBase, our data didn't have a record identifier, so I use the UpdateRecord processor to create one and add it to the data. I updated the schema to have this field (and have a default and allow nulls). As you can see it's pretty easy to store data to HBase. Schema: { "type" : "record", "name" : "drone", "fields" : [ { "name" : "drone_rec_id", "type" : [ "string", "null" ], "default": "1000" }, { "name" : "mvo_vel_x", "type" : ["double","null"], "default": "0.00" }, { "name" : "mvo_vel_y", "type" : ["string","null"], "default": "0.00" }, { "name" : "mvo_vel_z", "type" : ["double","null"], "default": "0.00" }, { "name" : "mvo_pos_x", "type" : ["string","null"], "default": "0.00" }, { "name" : "mvo_pos_y", "type" : ["double","null"], "default": "0.00"}, { "name" : "mvo_pos_z", "type" : ["string","null"], "default": "0.00" }, { "name" : "imu_acc_x", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_acc_y", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_acc_z", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_gyro_x", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_gyro_y", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_gyro_z", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_q0", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_q1", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_q2", "type" : ["double","null"], "default": "0.00" }, { "name" : "self_q3", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_vg_x", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_vg_y", "type" : ["double","null"], "default": "0.00" }, { "name" : "imu_vg_z", "type" : ["double","null"], "default": "0.00" } ] } The updated schema now has a record id. The original schema derived from the raw data does not. Store the Data in HBase Table Soon we will be storing in Kudu, Impala, Hive, Druid and S3. create 'drone', 'drone' Source: We are using the TelloPy interface. You need to clone this github and drop in the files from nifi-drone. https://github.com/hanyazou/TelloPy/ https://github.com/tspannhw/nifi-drone Apache NiFi Flows: dronelocal.xml dronecloud.xml References: https://github.com/hanyazou/TelloPy https://gobot.io/blog/2018/04/20/hello-tello-hacking-drones-with-go/ https://github.com/grofattila/dji-tello https://github.com/dbaldwin/droneblocks-tello-python https://medium.com/@makerhacks/programming-the-ryze-dji-tello-with-python-eecd56fc2c27 https://github.com/Ubotica/telloCV/ https://www.instructables.com/id/Ultimate-Intelligent-Fully-Automatic-Drone-Robot-w/ https://github.com/hybridgroup/gobot/tree/master/platforms/dji/tello https://www.ryzerobotics.com/tello https://dl-cdn.ryzerobotics.com/downloads/Tello/20180404/Tello_User_Manual_V1.2_EN.pdf https://dl-cdn.ryzerobotics.com/downloads/Tello/20180212/Tello+Quick+Start+Guide_V1.2+multi.pdf https://dl-cdn.ryzerobotics.com/downloads/tello/20180910/Tello%20Scratch%20README.pdf https://dl-cdn.ryzerobotics.com/downloads/tello/20180910/scratch0907.7z https://www.ryzerobotics.com/tello/downloads https://www.hackster.io/econnie323/alexa-voice-controlled-tello-drone-760615 https://tellopilots.com/forums/tello-development.8/ https://medium.com/@swalters/dji-ryze-tello-drone-gets-reverse-engineered-46a65d83e6b5 http://www.fabriziomarini.com/2018/04/java-udp-drone-tello.html?m=1 https://github.com/microlinux/tello/blob/master/tello.py https://github.com/hybridgroup/gophercon-2018/blob/master/drone/tello/README.md https://tellopilots.com/threads/object-tracking-with-tello.1480/ https://github.com/gnamingo/jTello/blob/master/JTello.java https://github.com/microlinux/tello/blob/master/README.md https://steemit.com/python/@makerhacks/programming-the-ryze-dji-tello-with-python https://github.com/hanyazou/TelloPy https://github.com/dji-sdk/Tello-Python https://github.com/Ubotica/telloCV/ https://github.com/dji-sdk/Tello-Python/tree/master/Tello_Video_With_Pose_Recognition https://github.com/DaWelter/h264decoder https://github.com/twilightdema/h264j http://jcodec.org/ https://github.com/cisco/openh264 https://github.com/hanyazou/TelloPy/blob/develop-0.7.0/tellopy/examples/video_effect.py https://gobot.io/blog/2018/04/20/hello-tello-hacking-drones-with-go/ https://medium.com/@swalters/dji-ryze-tello-drone-gets-reverse-engineered-46a65d83e6b5 ... View more

Implementing Streaming Machine Learning and Deep Learning In Production Part 1 After we have done our data exploration with Apache Zeppelin, Hortonworks Data Analytics Studio and other Data Science Notebooks and Tools, we will start building iterations of ever improving models that need to be used in live environments. These will need to run at scale and score millions of records in real-time streams. These can be in various frameworks, versions,types and many options of data required. There are a number of things we need to think about when doing this. Model Deployment Options Apache Spark Apache Storm (Hortonworks Streaming Analytics Manager - SAM) Apache Kafka Streams Apache NiFi YARN 3.1 YARN Submarine TensorFlow Serving on YARN Cloudera Data Science Workbench Requirements Classification REST API Security Automation Data Lineage Schema Versioning, REST API and Management Data Provenance Scripting Integration with Kafka Containerized Services Support Docker Containers running on YARN Support Dockerized Spark Jobs Model Registry Scalability Data Variety Data and Storage Format Flexiblity Handling Media Types such as images, sound and video Required Elements Apache NiFi 1.8.0 Apache Kafka 2.0 Apache Kafka Streams 2.0 Apache Atlas 1.0.0 Apache Ranger 1.2.0 Apache Knox 1.0 Hortonworks Streams Messaging Manager 1.2.0 Hortonworks Schema Registry 0.5.2 NiFi Registry 0.2.0 Apache Hadoop 3.1 Apache YARN 3.1+ Apache HDFS or Amazon S3 Apache Druid 0.12.1 Apache HBase 2.0 Apache Spark - Apache NiFi There are a number of options for running Machine Learning models in production via Apache NiFi. I have use these methods. Apache NiFi to Apache Spark Integration via Kafka and Spark Streaming Apache NiFi to Apache Spark Integration via Kafka and Spark Structured Streaming Apache NiFi to Apache Spark Integration via Apache Livy https://community.hortonworks.com/content/kbentry/174105/hdp-264-hdf-31-apache-spark-structured-streaming-i.html https://community.hortonworks.com/articles/174105/hdp-264-hdf-31-apache-spark-structured-streaming-i.html https://community.hortonworks.com/content/kbentry/171787/hdf-31-executing-apache-spark-via-executesparkinte.html Hadoop - YARN 3.1 - No Docker - No Spark We can deploy Deep Learning Models and run classification (as well as training) on YARN natively. https://community.hortonworks.com/content/kbentry/222242/running-apache-mxnet-deep-learning-on-yarn-31-hdp.html https://community.hortonworks.com/articles/224268/running-tensorflow-on-yarn-31-with-or-without-gpu.html Apache Kafka Streams Kafka Streams has full integration Platform services including Schema Registry, Ranger and Ambari. Apache NiFi Native Java Processors for Classification We can use a custom processor in Java that runs as a native part of the dataflow. https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html https://github.com/tspannhw/nifi-tensorflow-processor https://community.hortonworks.com/articles/229215/apache-nifi-processor-for-apache-mxnet-ssd-single.html https://github.com/tspannhw/nifi-mxnetinference-processor Apache NiFi Integration with a Model Server Native to a Framework Apache MXNet has an open source model server that has a full REST API that can easily be integrated with Apache NiFi. https://community.hortonworks.com/articles/155435/using-the-new-mxnet-model-server.html https://community.hortonworks.com/articles/223916/posting-images-with-apache-nifi-17-and-a-custom-pr.html https://community.hortonworks.com/articles/177232/apache-deep-learning-101-processing-apache-mxnet-m.html To run Apache MXNet model server is easy: mxnet-model-server --models SSD=resnet50_ssd_model.model --service ssd_service.py --port 9998 TensorFlow also has a model server that supports gRPC and REST. https://www.tensorflow.org/serving/api_rest Hortonworks Streaming Analytics Manager (SAM) SAM supports running machine learning models exported as PMML as part of a flow. https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/getting-started-with-streaming-analytics/content/export_the_model_into_sam%27s_model_registry.html https://hortonworks.com/blog/part-4-sams-stream-builder-building-complex-stream-analytics-apps-without-code/ You can score the model in a fully graphical manner: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/getting-started-with-streaming-analytics/content/score_the_model_using_the_pmml_processor_and_alert.html Deep Work on Model Governance and Integration with Apache Atlas: Customizing Atlas (Part1): Model governance, traceability and registry Generalized Framework to Deploy Models and Integrate Apache Atlas for Model Governance Customizing Atlas (Part2): Deep source metadata & embedded entities Customizing Atlas (Part3): Lineage beyond Hadoop, including reports & emails References: https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/68140 https://apachecon.dukecon.org/acna/2018/#/scheduledEvent/7058e0d4f5ab28836 https://dataworkssummit.com/berlin-2018/session/iot-with-apache-mxnet-and-apache-nifi-and-minifi/ https://dataworkssummit.com/berlin-2018/session/apache-deep-learning-101/ https://dataworkssummit.com/san-jose-2018/session/open-source-computer-vision-with-tensorflow-apache-minifi-apache-nifi-opencv-apache-tika-and-python-2/ https://www.slideshare.net/bunkertor/apache-deep-learning-201-philly-open-source https://www.slideshare.net/bunkertor/running-apache-nifi-with-apache-spark-integration-options ... View more

IoT Series: Sensors: Utilizing Breakout Garden Hat: Part 2 - Integrating MQTT, TensorFlow and Kafka Streams See Part 1: https://community.hortonworks.com/content/kbentry/229522/iot-series-sensors-utilizing-breakout-garden-hat-p.html In this second part, I have incremented the functionality in the Python capture, MiniFi, NiFi and post-NiFi processing. I have added a Kafka Streams Java application. With this NiFi flow we are consuming the MQTT and Kafka messages send by the Kafka Streams application. In one flow, we received MQTT messages, pull out the entire flow as a message and send to a Slack channel. In another flow we ingest two types of Kafka messages and store the JSON ones that have a schema in an HBase table via the record processor. In this flow we receive from the local NiFi router that was called by MiniFi over S2S/HTTP(s). We build two types of messages and send them to Kafka 2.0 brokers. One is the full JSON message with a schema, the other is just the temperature. We create a Kafka Key from the UUID. We also process the images sent from MiniFi with my native Java TensorFlow Inception processor. I decided to try some TensorFlow processing for our infinite sensor loop, it may be too much memory usage, so I may have to pick a different TensorFlow model and switch to TF Lite (https://www.tensorflow.org/lite/devguide). You will not two extra attributes coming from the Python script running on the Raspberry Pi 3B+. Another thing I wanted to do is try Kafka Streams since in Kafka 2.0 in HDP and HDF we have a fully supported version available. So based on example code I wrote a simple Kafka Streams Java 8 application that reads Kafka JSON messages sent from NiFi 1.8 and check for some conditions and push out data to MQTT and another Kafka topic. If you don't have an MQTT broker. Here is a quick way to install a Mosquitto MQTT broker on Centos 7. sudo yum -y install mosquitto /etc/mosquitto/mosquitto.conf mkdir -p /var/log/mosquitto<br>chmod -R 777 /var/log/mosquitto/<br>touch /var/log/mosquitto/mosquitto.log<br>sudo systemctl start mosquitto<br>sudo systemctl enable mosquitto Now that we have an MQTT broker our Kafka Streams app can send messages to it and NiFi can read messages from it. In a future version I will use Hortonworks Schema Registry and Avro. I have updated the Python script to include TensorFlow and to update to Python 3.5. Make sure you run with Python 3.5 and have all the libraries installed on your RPI/Linux device. Some of the updated code for 3.5, note the message encoding. Python: https://github.com/tspannhw/minifi-breakoutgarden/blob/master/minifi35.py def send_tcp(s, message): if not message: try: s.sendall(message.encode('utf-8')) <br> except: print("Failed to send message") For testing IOT values, I have a GenerateFlowFile with this JSON: { "systemtime" : "${now():format('MM/dd/yyyy HH:mm:ss')}", "BH1745_green" : "${random():mod(100):plus(1)} ", "ltr559_prox" : "0000", "end" : "${now():format('yyyyMMddHHmmss')}", "uuid" : "${now():format('yyyyMMddHHmmss')}_${UUID()}", "lsm303d_accelerometer" : "+00.06g : -01.01g : +00.04g", "imgnamep" : "images/bog_image_p_${now():format('yyyyMMddHHmmss')}_${UUID()}.jpg", "cputemp" : ${random():mod(100):toNumber()}, "BH1745_blue" : "9.0", "te" : "47.3427119255", "bme680_tempc" : "28.19", "imgname" : "images/bog_image_${now():format('yyyyMMddHHmmss')}_${UUID()}.jpg", "bme680_tempf" : "80.${random():mod(100):toNumber()}", "ltr559_lux" : "006.87", "memory" : 34.9, "VL53L1X_distance_in_mm" : 134, "bme680_humidity" : "${random():mod(100):toNumber()}", "host" : "vid5", "diskusage" : "8732.7", "ipaddress" : "192.168.1.167", "bme680_pressure" : "1017.31", "BH1745_clear" : "10.0", "BH1745_red" : "0.0", "lsm303d_magnetometer" : "+00.04 : +00.34 : -00.10", "starttime" : "${now():format('MM/dd/yyyy HH:mm:ss')}" } Kafka Streams Source Code: https://github.com/tspannhw/kstreams Running the Fat Jar: java -jar target/kstreams-1.0.jar<br>******************************************* Started <br>**********2018/12/28 16:41:19<br>********** Memory Usage: 28284968 Updated Source Code: https://github.com/tspannhw/minifi-breakoutgarden Updated Example Run Output { "ltr559_lux" : "033.75", "uuid" : "20181228162321_cbd0cbd3-17f6-4730-ae43-1e7b46a01135", "cputemp" : 51, "host" : "piups", "lsm303d_magnetometer" : "-00.12 : +00.27 : +00.15", "bme680_tempc" : "24.96", "score" : "0.9694475", "lsm303d_accelerometer" : "+00.12g : -01.00g : +00.08g", "ltr559_prox" : "0000", "bme680_humidity" : "28.875", "diskusage" : "10058.7", "human_string" : "electric fan, blower", "bme680_pressure" : "1012.00", "BH1745_green" : "31.0", "imgnamep" : "/opt/demo/images/bog_image_p_20181228162321_cbd0cbd3-17f6-4730-ae43-1e7b46a01135.jpg", "systemtime" : "12/28/2018 11:24:11", "BH1745_red" : "33.0", "starttime" : "12/28/2018 11:16:02", "BH1745_blue" : "19.8", "end" : "1546014251.2879872", "bme680_tempf" : "76.93", "VL53L1X_distance_in_mm" : 455, "te" : "488.33915853500366", "memory" : 70.8, "imgname" : "/opt/demo/images/bog_image_20181228162321_cbd0cbd3-17f6-4730-ae43-1e7b46a01135.jpg", "ipaddress" : "192.168.1.166", "BH1745_clear" : "40.0" } From Kafka Streams I am sending a warning on temperature to MQTT which NiFi sends to Slack. Temperature warning 82.74 Using HBase 2.0, we are storing out data as it streams from Kafka Streams to NiFi. We use PutHBaseRecord which utilizes record processing and our schema to stream our JSON into HBase with ease. Updated Schema with TF Attributes { "type": "record", "name": "garden", "fields": [ { "name": "systemtime", "type": "string" }, { "name": "BH1745_green", "type": "string" }, { "name": "human_string", "type": "string", "default": "UNK" }, { "name": "ltr559_prox", "type": "string" }, { "name": "end", "type": "string" }, { "name": "uuid", "type": "string" }, { "name": "lsm303d_accelerometer", "type": "string" }, { "name": "score", "type": "string", "default": "0" }, { "name": "imgnamep", "type": "string" }, { "name": "cputemp", "type": "double", "doc": "Type inferred from '58.0'" }, { "name": "BH1745_blue", "type": "string", "doc": "Type inferred from '\"10.8\"'" }, { "name": "te", "type": "string", "doc": "Type inferred from '\"254.545491934\"'" }, { "name": "bme680_tempc", "type": "string", "doc": "Type inferred from '\"29.13\"'" }, { "name": "imgname", "type": "string" }, { "name": "bme680_tempf", "type": "string", "doc": "Type inferred from '\"84.43\"'" }, { "name": "ltr559_lux", "type": "string", "doc": "Type inferred from '\"077.95\"'" }, { "name": "memory", "type": "double", "doc": "Type inferred from '37.6'" }, { "name": "VL53L1X_distance_in_mm", "type": "int", "doc": "Type inferred from '161'" }, { "name": "bme680_humidity", "type": "string", "doc": "Type inferred from '\"32.359\"'" }, { "name": "host", "type": "string", "doc": "Type inferred from '\"vid5\"'" }, { "name": "diskusage", "type": "string", "doc": "Type inferred from '\"8357.6\"'" }, { "name": "ipaddress", "type": "string", "doc": "Type inferred from '\"192.168.1.167\"'" }, { "name": "bme680_pressure", "type": "string", "doc": "Type inferred from '\"987.86\"'" }, { "name": "BH1745_clear", "type": "string", "doc": "Type inferred from '\"90.0\"'" }, { "name": "BH1745_red", "type": "string", "doc": "Type inferred from '\"33.0\"'" }, { "name": "lsm303d_magnetometer", "type": "string" }, { "name": "starttime", "type": "string" } ] } HBase table create 'breakout', 'sensors' Example Row 1546014251.2879872 column=sensors:BH1745_blue, timestamp=1546020326955, value=19.8 1546014251.2879872 column=sensors:BH1745_clear, timestamp=1546020326955, value=40.0 1546014251.2879872 column=sensors:BH1745_green, timestamp=1546020326955, value=31.0 1546014251.2879872 column=sensors:BH1745_red, timestamp=1546020326955, value=33.0 1546014251.2879872 column=sensors:VL53L1X_distance_in_mm, timestamp=1546020326955, value=455 1546014251.2879872 column=sensors:bme680_humidity, timestamp=1546020326955, value=28.875 1546014251.2879872 column=sensors:bme680_pressure, timestamp=1546020326955, value=1012.00 1546014251.2879872 column=sensors:bme680_tempc, timestamp=1546020326955, value=24.96 1546014251.2879872 column=sensors:bme680_tempf, timestamp=1546020326955, value=76.93 1546014251.2879872 column=sensors:cputemp, timestamp=1546020326955, value=51.0 1546014251.2879872 column=sensors:diskusage, timestamp=1546020326955, value=10058.7 1546014251.2879872 column=sensors:host, timestamp=1546020326955, value=piups 1546014251.2879872 column=sensors:human_string, timestamp=1546020326955, value=electric fan, blower 1546014251.2879872 column=sensors:imgname, timestamp=1546020326955, value=/opt/demo/images/bog_image_201812281 62321_cbd0cbd3-17f6-4730-ae43-1e7b46a01135.jpg 1546014251.2879872 column=sensors:imgnamep, timestamp=1546020326955, value=/opt/demo/images/bog_image_p_201812 28162321_cbd0cbd3-17f6-4730-ae43-1e7b46a01135.jpg 1546014251.2879872 column=sensors:ipaddress, timestamp=1546020326955, value=192.168.1.166 1546014251.2879872 column=sensors:lsm303d_accelerometer, timestamp=1546020326955, value=+00.12g : -01.00g : +0 0.08g 1546014251.2879872 column=sensors:lsm303d_magnetometer, timestamp=1546020326955, value=-00.12 : +00.27 : +00.1 5 1546014251.2879872 column=sensors:ltr559_lux, timestamp=1546020326955, value=033.75 1546014251.2879872 column=sensors:ltr559_prox, timestamp=1546020326955, value=0000 1546014251.2879872 column=sensors:memory, timestamp=1546020326955, value=70.8 1546014251.2879872 column=sensors:score, timestamp=1546020326955, value=0.9694475 1546014251.2879872 column=sensors:starttime, timestamp=1546020326955, value=12/28/2018 11:16:02 1546014251.2879872 column=sensors:systemtime, timestamp=1546020326955, value=12/28/2018 11:24:11 1546014251.2879872 column=sensors:te, timestamp=1546020326955, value=488.33915853500366 1546014251.2879872 column=sensors:uuid, timestamp=1546020326955, value=20181228162321_cbd0cbd3-17f6-4730-ae43- 1e7b46a01135 ... View more

First off, this was an amazing year for Big Data, IoT, Kafka, AI, Streaming, Machine Learning and Deep Learning. So many cool events, updates, new products, new projects, new libraries and community growth. I've seen a lot of people adopt and grow Big Data and streaming projects from nothing. Using the power of Open Source and the tools made available by Apache, companies are growing with the help of trusted partners and a community of engineers and users. I had a fun year travelling to various conferences showing how to use Open Source technologies for IoT, Deep Learning, Machine Learning, Streaming, Big Data, Computer Vision and microservices. First up close to home was Iot Fusion in Philly: http://iotfusion.net/session/enterprise-iiot-edge-processing-with-apache-nifi-minifi-and-deep-learning/ It was a very cool event run by my friends at Chariot Solutions. We focused on some great technology from 2018 including MiniFi, Apache NiFi, TensorFlow and Apache MXNet. After that I headed to Europe and spoke at an awesome meetup in Prague. https://www.meetup.com/futureofdata-prague/events/248994531/ https://www.slideshare.net/bunkertor/deep-learning-on-hdp-2018-prague. A short train ride up to Berlin for a meetup https://www.meetup.com/futureofdata-berlin/events/249190513/ https://www.slideshare.net/bunkertor/minifi-and-apache-nifi-iot-in-berlin-germany-2018. Then I was in Berlin for Dataworks Summit and did two talks. https://dataworkssummit.com/berlin-2018/session/iot-with-apache-mxnet-and-apache-nifi-and-minifi/ and https://dataworkssummit.com/session/apache-deep-learning-101/. https://dzone.com/articles/apache-nifi-at-dws In June it was off to San Jose for the Big DataWorks Summit and I had a talk on Computer Vision. https://dataworkssummit.com/san-jose-2018/session/open-source-computer-vision-with-tensorflow-apache-minifi-apache-nifi-opencv-apache-tika-and-python-2/ https://www.slideshare.net/Hadoop_Summit/open-source-computer-vision-with-tensorflow-apache-minifi-apache-nifi-opencv-apache-tika-and-python I also helped out with some TensorFlow in Apache Zeppelin at the Deep Learning Crash Course https://dataworkssummit.com/san-jose-2018/crash-courses/. Come August, I joined the IoT Webinar https://hortonworks.com/webinar/iot-war-stories-challenges-solutions-best-practices/ which was a lot of fun and then got to speak with the awesome Data Scientist, John Kuchmek at the NJ Shore. Fall quickly followed and it was time for Strata in NYC. https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/speaker/185963 Again I spoke on IoT with AI https://github.com/tspannhw/StrataNYC2018 After Strata it was a quick flight up to Montreal for ApacheCon. It was great seeing all the Apache speakers and my friends from Apache MXNet. https://apachecon.dukecon.org/acna/2018/#/scheduledEvent/7058e0d4f5ab28836 https://feathercast.apache.org/2018/10/02/apache-deep-learning-101-timothy-spann/ In October I was down in Orlando with my AI buddies AppOrchid to speak about AI, IoT and Big Data for Utilities. http://www.utilityanalyticsweek.com/cr3ativspeaker/tim-spann/ https://www.slideshare.net/bunkertor/the-best-of-both-worlds-delivering-digital-transformation In November I got to be the first speaker at Philadelphia's first Open Source Conference hosted by Comcast! https://phillyopensource.splashthat.com/ What an amazing event and awesome people running this. Hats off to the Comcast team for support Open Source and having great projects. They also were big sponsors of Apache Con. To round it out, in December I reunited with my AI friends on a webinar for utilities https://utilityanalytics.com/webinar/providing-ai-solutions-to-break-the-data-silos-and-accelerate-utilities-into-the-forefront-of-data-driven-decisions/. The year was great for meetups and I got to talk at a few around the world, plus Milind and my meetup in Princeton grew to 1,100 members! Check us out: https://www.meetup.com/futureofdata-princeton/ My friend Paul restarted the https://www.meetup.com/futureofdata-philadelphia/ meetup and we will do some fun stuff in 2019 around the new Camden tech region. Nov 14 - Rafi from IBM https://www.meetup.com/futureofdata-princeton/events/254821251/ Oct 2nd - Mehul ( https://infinity-services.com/) and Abdhul (http://cloudsegue.com/) spoke. Hosted by TRAC Intermodal around Blockchain. https://www.meetup.com/futureofdata-princeton/events/252716511/ Sept 6 - Thomas spoke on MXNet https://www.meetup.com/futureofdata-princeton/events/253312744/ June 28 - I spoke on NiFi and Blockchain https://www.meetup.com/futureofdata-princeton/events/249163765/ Thanks to Prasad (Airisdata). Joint meetup in Hamilton with the NJ_Blockchain meetup. May 8 - I spoke on NiFi and IoT in NYC https://www.meetup.com/futureofdata-princeton/events/250304839/ April 17 - I spoke on NiFi in Berlin https://www.meetup.com/futureofdata-berlin/events/249190513/ April 12 - I spoke on NiFi and Deep Learning in Prague FoD https://www.meetup.com/futureofdata-prague/events/248994531/ Feb 13 - Big Mardis Gras celebration at TRAC Intermodal We have a meetup scheduled for January on Blockchain in Woodbridge. ChainNinja is going to broadcast this: https://www.meetup.com/futureofdata-princeton/events/255404291/ From meetups, webinars, workshops and conferences I had a few presentations: https://www.slideshare.net/bunkertor/machine-learning-and-deep-learning-on-hdp-301-and-hdf-32 https://www.slideshare.net/bunkertor/apache-deep-learning-201-philly-open-source https://www.slideshare.net/bunkertor/the-best-of-both-worlds-delivering-digital-transformation https://www.slideshare.net/bunkertor/apache-deep-learning-101-apachecon-montreal-2018-v031 https://www.slideshare.net/bunkertor/handson-deep-dive-with-minifi-and-apache-mxnet https://www.slideshare.net/bunkertor/open-source-predictive-analytics-pipeline-with-apache-nifi-and-minifi-princeton https://www.slideshare.net/bunkertor/open-computer-vision-with-opencv-apache-nifi-tensorflow-python https://www.slideshare.net/bunkertor/iot-edge-processing-with-apache-nifi-and-minifi-and-apache-mxnet-for-iot-ny-2018 https://www.slideshare.net/bunkertor/apache-mxnet-for-iot-with-apache-nifi https://www.slideshare.net/bunkertor/apache-deep-learning-101-dws-berlin-2018 https://www.slideshare.net/bunkertor/minifi-and-apache-nifi-iot-in-berlin-germany-2018 https://www.slideshare.net/bunkertor/deep-learning-on-hdp-2018-prague https://www.slideshare.net/bunkertor/enterprise-iiot-edge-processing-with-apache-nifi-92970386 https://www.slideshare.net/bunkertor/running-apache-nifi-with-apache-spark-integration-options https://www.slideshare.net/bunkertor/hdf-31-an-introduction-to-new-features I wrote a few articles for HCC in 2018: https://community.hortonworks.com/articles/167193/building-and-running-minifi-cpp-in-orangepi-zero.html https://community.hortonworks.com/articles/171787/hdf-31-executing-apache-spark-via-executesparkinte.html https://community.hortonworks.com/articles/171893/hdf-31-executing-apache-spark-via-executesparkinte-1.html https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an-apache-nifi-15-instance-w.html https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html https://community.hortonworks.com/articles/174399/apache-deep-learning-101-using-apache-mxnet-on-apa.html https://community.hortonworks.com/articles/176784/deep-learning-101-using-apache-mxnet-in-dsx-notebo.html https://community.hortonworks.com/articles/176789/apache-deep-learning-101-using-apache-mxnet-in-apa.html https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on-the.html https://community.hortonworks.com/articles/177232/apache-deep-learning-101-processing-apache-mxnet-m.html https://community.hortonworks.com/articles/177663/apache-livy-apache-nifi-apache-spark-executing-sca.html https://community.hortonworks.com/articles/178498/integrating-tensorflow-16-image-labelling-with-hdf.html https://community.hortonworks.com/articles/183806/using-a-tensorflow-person-blocker-with-apache-nifi.html https://community.hortonworks.com/articles/198912/ingesting-apache-mxnet-gluon-deep-learning-results.html https://community.hortonworks.com/articles/222242/running-apache-mxnet-deep-learning-on-yarn-31-hdp.html https://community.hortonworks.com/articles/193868/integrating-keras-tensorflow-yolov3-into-apache-ni.html https://community.hortonworks.com/articles/198855/executing-tensorflow-classifications-from-apache-n.html https://community.hortonworks.com/articles/198939/using-apache-mxnet-gluoncv-with-apache-nifi-for-de.html https://community.hortonworks.com/articles/215079/iot-edge-processing-with-deep-learning-on-hdf-32-a.html https://community.hortonworks.com/articles/224268/running-tensorflow-on-yarn-31-with-or-without-gpu.html https://community.hortonworks.com/articles/203638/ingesting-multiple-iot-devices-with-apache-nifi-17.html https://community.hortonworks.com/articles/227194/ingesting-and-analyzing-street-camera-data-from-ma.html https://community.hortonworks.com/articles/207858/more-devops-for-hdf-apache-nifi-and-friends.html https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html https://community.hortonworks.com/articles/167187/provenance-site-to-site-reporting.html https://community.hortonworks.com/articles/177370/extracting-html-from-pdf-excel-and-word-documents.html https://community.hortonworks.com/articles/177733/apache-nifi-processor-building-a-sql-ddl-schema-fr.html https://community.hortonworks.com/articles/192848/updating-the-apache-opennlp-community-apache-nifi.html https://community.hortonworks.com/articles/193822/parsing-web-pages-for-images-with-apache-nifi.html https://community.hortonworks.com/articles/193835/detecting-language-with-apache-nifi.html https://community.hortonworks.com/articles/189213/etl-with-lookups-with-apache-hbase-and-apache-nifi.html https://community.hortonworks.com/articles/222605/converting-powerpoint-presentations-into-french-fr.html https://community.hortonworks.com/articles/223840/properties-file-lookup-augmentation-of-data-flow-i.html https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html https://community.hortonworks.com/articles/155606/iot-ingesting-gps-data-from-odroid-xu4-devices-wit.html https://community.hortonworks.com/articles/161761/new-features-in-apache-nifi-15-apache-nifi-registr.html https://community.hortonworks.com/articles/167196/ingesting-data-from-the-matrix-creator-with-minifi.html https://community.hortonworks.com/articles/167199/sending-messages-and-displaying-them-on-an-oled-sc.html https://community.hortonworks.com/articles/173818/hdp-264-hdf-31-apache-spark-streaming-integration.html https://community.hortonworks.com/articles/174105/hdp-264-hdf-31-apache-spark-structured-streaming-i.html https://community.hortonworks.com/articles/174538/apache-deep-learning-101-using-apache-mxnet-with-h.html https://community.hortonworks.com/articles/177137/ingesting-flight-data-ads-b-usb-receiver-with-apac.html https://community.hortonworks.com/articles/177256/spring-boot-20-on-acid-integrating-rest-microservi.html https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au.html https://community.hortonworks.com/articles/178196/integrating-lucene-geo-gazetteer-for-geo-parsing-w.html https://community.hortonworks.com/articles/178510/integration-apache-opennlp-184-into-apache-nifi-15.html https://community.hortonworks.com/articles/182850/vision-thing.html https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html https://community.hortonworks.com/articles/183217/devops-backing-up-apache-nifi-registry-flows.html https://community.hortonworks.com/articles/183474/iot-using-minifi-java-agent-to-send-mqtt-messages.html https://community.hortonworks.com/articles/185079/publishing-and-consuming-jms-messages-from-tibco-e.html https://community.hortonworks.com/articles/189514/converting-csv-files-to-apache-hive-tables-with-ap.html https://community.hortonworks.com/articles/189630/tracking-air-quality-with-hdp-and-hdfi-part-1-apac.html https://community.hortonworks.com/articles/189735/automating-social-media-sending-tweets-with-apache.html https://community.hortonworks.com/articles/190765/processing-real-time-social-media-twitter-with-apa.html https://community.hortonworks.com/articles/191146/accessing-feeds-from-etherdelta-on-trades-funds-bu.html https://community.hortonworks.com/articles/191255/ethereum-accessing-feeds-from-etherscan-on-volume.html https://community.hortonworks.com/articles/191658/devops-tips-using-the-apache-nifi-toolkit-with-apa.html https://community.hortonworks.com/articles/196963/scanning-documents-into-data-lakes-via-tesseract-p.html https://community.hortonworks.com/articles/199566/ingesting-infura-rest-apis-to-access-the-ethereum.html https://community.hortonworks.com/articles/199570/ingest-btccom-and-blockchaincom-data-via-apache-ni.html https://community.hortonworks.com/articles/219777/iot-edge-processing-with-deep-learning-on-hdf-32-a-3.html https://community.hortonworks.com/articles/215271/iot-edge-processing-with-deep-learning-on-hdf-32-a-2.html https://community.hortonworks.com/articles/224556/building-a-custom-apache-nifi-operations-dashboard-1.html https://community.hortonworks.com/articles/224554/building-a-custom-apache-nifi-operations-dashboard.html https://community.hortonworks.com/articles/222367/using-apache-nifi-with-apache-mxnet-gluoncv-for-yo.html https://community.hortonworks.com/articles/223916/posting-images-with-apache-nifi-17-and-a-custom-pr.html https://community.hortonworks.com/articles/202236/integrating-apache-mxnet-model-server-with-apache.html https://community.hortonworks.com/articles/191259/integrating-darknet-yolov3-into-apache-nifi-workfl.html https://community.hortonworks.com/articles/215258/iot-edge-processing-with-deep-learning-on-hdf-32-a-1.html https://community.hortonworks.com/articles/177349/big-data-devops-apache-nifi-hwx-schema-registry-sc.html https://community.hortonworks.com/articles/227560/real-time-stock-processing-with-apache-nifi-and-ap.html https://community.hortonworks.com/articles/228874/iot-edge-use-cases-with-apache-kafka-and-apache-ni.html https://community.hortonworks.com/content/kbentry/229215/apache-nifi-processor-for-apache-mxnet-ssd-single.html https://community.hortonworks.com/articles/229305/using-apache-nifi-for-speech-processing-speech-to.html https://community.hortonworks.com/articles/229522/iot-series-sensors-utilizing-breakout-garden-hat-p.html https://community.hortonworks.com/articles/232136/iot-series-sensors-utilizing-breakout-garden-hat-p-1.html I wrote a few custom processors in Java for Apache NiFi (all work in NiFi 1.8.0): https://github.com/tspannhw/nifi-mxnetinference-processor https://github.com/tspannhw/nifi-extracttext-processor https://github.com/tspannhw/nifi-langdetect-processor https://github.com/tspannhw/nifi-attributecleaner-processor https://github.com/tspannhw/nifi-convertjsontoddl-processor https://github.com/tspannhw/nifi-postimage-processor https://github.com/tspannhw/GetWebCamera https://github.com/tspannhw/nifi-imageextractor-processor https://github.com/tspannhw/nifi-puttwitter-processor https://github.com/tspannhw/nifi-tensorflow-processor A wrote a few other scripts and utilities outside of NiFi as well: https://github.com/tspannhw/nifi-registry-github https://github.com/tspannhw/python-scripts https://github.com/tspannhw/operations-dashboard https://github.com/tspannhw/stocks-nifi-kafka https://github.com/tspannhw/ApacheDeepLearning201 https://github.com/tspannhw/nifi-smartplug https://github.com/tspannhw/yolo3-keras-tensorflow https://github.com/tspannhw/TensorflowOnYARN https://github.com/tspannhw/nifi-gluoncv-yolo3 https://github.com/tspannhw/ApacheDeepLearning101 https://github.com/tspannhw/StrataNYC2018 https://github.com/tspannhw/UsingGluonCV https://github.com/tspannhw/OpenSourceComputerVision https://github.com/tspannhw/DWS-DeepLearning-CrashCourse https://github.com/tspannhw/IoTFusion2018Talk https://github.com/tspannhw/nifi-tesseract-python https://github.com/tspannhw/nifi-yolo3 https://github.com/tspannhw/mxnet-for-iot https://github.com/tspannhw/livysparkjob I started to do some video content in 2018, look for more of this than you will ever want to watch coming in 2019. https://www.youtube.com/watch?v=u4NZHBDyf54&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=5&t=10s https://www.youtube.com/watch?v=5w6rV7562xM&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=7&t=1497s https://www.youtube.com/watch?v=bOfSnNVum_M&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=8&t=409s https://www.youtube.com/watch?v=ksDKNp6Z4BE&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=9&t=39s https://www.youtube.com/watch?v=J23wgoIknP0&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=10&t=0s https://www.youtube.com/watch?v=uU9HO3SWbOs&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=11&t=158s https://www.youtube.com/watch?v=N0NLJo5y7RQ&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=13&t=1478s https://www.youtube.com/watch?v=Q4dSGPvqXSA&list=PL-7XqvSmQqfTEtNbnITDrgycfn3uny8LM&index=16&t=0s The Technology of 2018 That I most Used Apache NiFi Apache Kafka NiFi Registry MiniFi Apache Spark Hortonworks SMM Hortonworks Data Plane Services Apache MXNet TensorFlow Hadoop 3.1 HDP 3.1 and HDF 3.3 Apache Hive Apache Ranger Apache HBase Apache Phoenix Apache Druid Spring Boot Python 3.7 YOLO v3 MXNet GluonCV Kubernetes Hybrid Cloud Blockchain and Cryptocurrencies For a quick spin of the best of 2018, see here: https://community.hortonworks.com/articles/151939/hdp-securitygovernance-demo-kit.html I'll see you in 2019 as part of Cloudera with lots of new IoT, devices, AI, Deep Learning, NiFi, Kafka, Streaming and more. !!! Happy Holidays !!! References 2017: https://community.hortonworks.com/content/kbentry/155338/my-year-in-review-2017.html https://dzone.com/articles/dataworks-summit-2018-berlin-apache-nifi-wrapup https://dzone.com/articles/2018-the-year-in-big-data ... View more

IoT Series: Sensors: Utilizing Breakout Garden Hat: Part 1 - Introduction An easy option for adding, removing and prototype sensor reads from a standard Raspberry Pi with no special wiring. Hardware Component List: Raspberry Pi USB Power Cable Pimoroni Breakout Garden Hat 1.12" Mono OLED Breakout 128x128 White/Black Screen BME680 Air Quality, Temperature, Pressure, Humidity Sensor LWM303D 6D0F Motion Sensor (X, Y, Z Axes) BH1745 Luminance and Color Sensor LTR-559 Light and Proximity Sensor 0.01 lux to 64,000 lux VL53L1X Time of Flight (TOF) Sensor Pew Pew Lasers! Software Component List: Raspian Python 2.7 JDK 8 Java Apache NiFi MiniFi Source Code: https://github.com/tspannhw/minifi-breakoutgarden Shell Script (https://github.com/tspannhw/minifi-breakoutgarden/blob/master/runbrk.sh) Python (https://github.com/tspannhw/minifi-breakoutgarden/blob/master/brk.py) Summary Our Raspberry Pi has a Breakout Garden Hat with 5 sensors and one small display. The display is showing the last reading and is constantly updating. For debugging purposes, it shows the IP Address so I can connect as needed. We currently run via nohup, but when we go into constant use I will switch to a Linux Service to run on startup. The Python script initializes the connections to all of the sensors and then goes into an infinite loop of reading those values and building a JSON packet that we send via TCP/IP over port 5005 to a listener. MiniFi 0.5.0 Java Agent is using ListenTCP on that port to capture these messages and filter them based on alarm values. If outside of the checked parameters we send them via S2S/HTTP(s) to an Apache NiFi server. We also have a USB WebCam (Sony Playstation 3 EYE) that is capturing images and we read those with MiniFi and send them to NiFi as well. The first thing we need to do is pretty easy. We need to plug in our Pimoroni Breakout Garden Hat and our 6 plugs. You have to do the standard installation of Python, Java 8, MiniFi and I recommend OpenCV. Make sure you have everything plugged in securely and the correct direction before you power on the Raspberry Pi. Download MiniFi Java Here: https://nifi.apache.org/minifi/download.html Install Python PIP curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py Install Breakout Garden Library wget https://github.com/pimoroni/breakout-garden/archive/master.zip unzip master.zip cd breakout-garden-master sudo ./install.sh Running In NiFi First we build our MiniFi Flow: We have two objectives: listen for TCP/IP JSON messages from our running Python sensor collector and gather images captured by the PS3 Eye USB Webcam. We then add content type and schema information to the attributes. We also extract a few values from the JSON stream to use for alerting. We extract: $.cputemp, $.VL53L1X_distance_in_mm, $.bme680_humidity, $.bme680_tempf These attributes are now attached to our flowfile which is unchanged. We can now Route on them. So we route on a few alarm conditions: ${cputemp:gt(100)} ${humidity:gt(60)} ${tempf:gt(80)} ${distance:gt(0)} We can easily add more conditions or different set values. We can also populate these set values from an HTTP / file lookup. If these values are met we send to our local Apache NiFi router. This can then do further analysis with the fuller NiFi processor set including TensorFlow, MXNet, Record processing and lookups. Local NiFi Routing For now we are just splitting up the images and JSON and sending to two different remote ports on our cloud NiFi cluster. These then arrive in the cloud. As you can see a list of the flow files waiting to be processed (I haven't written that part yet). As you can see we are getting a few a second, we could get 100,000 a second if we needed. Just add nodes. Instant scaling. Cloudbreak can do that for you. In part 2, we will start processing these data streams and images. We will also add Apache MXNet and TensorFlow at various points on the edge, router and cloud using Python and built-in Deep Learning NiFi processors I have authored. We will also break apart these records and send each sensor to it's own Kafka topic to be processed with Kafka Streams, Druid, Hive and HBase. As part of our loop we write to our little screen current values: Example Record { "systemtime" : "12/19/2018 22:15:56", "BH1745_green" : "4.0", "ltr559_prox" : "0000", "end" : "1545275756.7", "uuid" : "20181220031556_e54721d6-6110-40a6-aa5c-72dbd8a8dcb2", "lsm303d_accelerometer" : "+00.06g : -01.01g : +00.04g", "imgnamep" : "images/bog_image_p_20181220031556_e54721d6-6110-40a6-aa5c-72dbd8a8dcb2.jpg", "cputemp" : 51.0, "BH1745_blue" : "9.0", "te" : "47.3427119255", "bme680_tempc" : "28.19", "imgname" : "images/bog_image_20181220031556_e54721d6-6110-40a6-aa5c-72dbd8a8dcb2.jpg", "bme680_tempf" : "82.74", "ltr559_lux" : "006.87", "memory" : 34.9, "VL53L1X_distance_in_mm" : 134, "bme680_humidity" : "23.938", "host" : "vid5", "diskusage" : "8732.7", "ipaddress" : "192.168.1.167", "bme680_pressure" : "1017.31", "BH1745_clear" : "10.0", "BH1745_red" : "0.0", "lsm303d_magnetometer" : "+00.04 : +00.34 : -00.10", "starttime" : "12/19/2018 22:15:09" } NiFi Templates nifi-garden-router.xml minifi-garden.xml garden-server.xml Let's Build Those Topics Now /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bme680 /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bh17455 /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic lsm303d /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic vl53l1x /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic ltr559 Hopefully in your environment, you will be able to have 3, 5 or 7 replication factor and many partitions. I have one Kafka Broker so this is what we are starting with. Reference https://shop.pimoroni.com/collections/breakout-garden https://github.com/pimoroni/breakout-garden/tree/master/examples https://shop.pimoroni.com/products/breakout-garden-hat https://github.com/pimoroni/bme680-python https://learn.pimoroni.com/bme680 https://shop.pimoroni.com/products/bh1745-luminance-and-colour-sensor-breakout https://github.com/pimoroni/bh1745-python https://shop.pimoroni.com/products/vl53l1x-breakout https://github.com/pimoroni/vl53l1x-python/tree/master/examples https://shop.pimoroni.com/products/ltr-559-light-proximity-sensor-breakout https://github.com/pimoroni/ltr559-python https://shop.pimoroni.com/products/bme680-breakout https://github.com/pimoroni/bme680 https://shop.pimoroni.com/products/lsm303d-6dof-motion-sensor-breakout https://github.com/pimoroni/lsm303d-python https://shop.pimoroni.com/products/1-12-oled-breakout https://github.com/rm-hull/luma.oled http://www.diegoacuna.me/how-to-run-a-script-as-a-service-in-raspberry-pi-raspbian-jessie/ ... View more

My Example Mesh Network Argon: Wi-Fi + Mesh Gateway/Repeater This is to connect to the outside network via WiFi. We connect to Particle Cloud. Espressif ESP32-D0WD 2.4G Wi-Fi Nordic Semiconductor nRF52840 SoC for bluetooth and NFC A tag ARM TrustZone CryptoCell-310 Cryptographic and security module Thread/BLE, another for Wi-Fi) Xenon: Mesh + BLE Nordic Semiconductor nRF52840 SoC for bluetooth and NFC A tag ARM TrustZone CryptoCell-310 Cryptographic and security module References https://github.com/Seeed-Studio/Grove_Starter_Kit_for_Photon_Demos?files=1 https://docs.particle.io/datasheets/accessories/mesh-accessories/ https://community.particle.io/c/mesh https://www.particle.io/mesh https://docs.particle.io/datasheets/mesh/xenon-datasheet/ https://docs.particle.io/datasheets/wi-fi/argon-datasheet/ https://blog.particle.io/2018/04/28/how-to-build-a-wireless-mesh-network/ WiFi Mesh = https://www.threadgroup.org/ https://github.com/Seeed-Studio/Grove_Starter_Kit_for_Photon_Demos/blob/master/Example%20-%2005%20Measuring%20Temperature/Example05.ino http://wiki.seeedstudio.com/Grove-Ultrasonic_Ranger/ https://www.particle.io/mesh/buy/xenon ... View more