Ingest All The Things Series: Flight Data Via Radio I am using the FlightAware Pro Stick Plus ADS-B USB Receiver with Built-in Filter on a Mac, I should hook this up to one of my raspberry pis and add a longer antenna outside. You need a good antenna, a good location and nothing blocking your signal. It also depends on what air traffic is nearby. For a proof of concept it's pretty cool to see air data going through a cheap USB stick into a computer stored in a file and loaded into Apache NiFi to send on for data processing. There is a web server you can run to see the planes on a map which is pretty cool, but I want to just ingest the data for processing. My Equipment If you wish to watch the data flash by in a command-line interface, you can run with interactive flag and watch all the updates. We are dumping the data as it streams as raw text into a file. A snippet of it tailed in Apache NiFi is shown below: We are also ingesting ADS-B data that is provided by an open data REST API (https://public-api.adsbexchange.com..) at https://www.adsbexchange.com/. Like everything else, we may want to add a schema to parse into records. Our ingest flow: I am getting the REST data from the ADSB Exchange REST API, tailing the raw text dump from dump1090 and reading the aircraft history JSON files produced by dump1090 as well. For further processing, I send the Aircraft History JSON files to my server cluster to send to a cloud hosted MongoDB database. Thanks to a free tier from mLab. And our data quickly arrives as JSON in Mongo. The main install is the dump1090 github and is pretty straight forward. Installation on OSX brew update brew install librtlsdr pkg-config make Running ./run2.sh >> raw.txt 2>&1 run2.sh ./dump1090 --net --lat 40.265887 --lon -74.534610 --modeac --mlat --write-json-every 1 --json-location-accuracy 2 --write-json /volumes/seagate/projects/dump1090/data I have entered my local latitude and longitude above. I also write to a local directory that we will read from in Apache NiFi. Example Data { "now" : 1507344034.5, "messages" : 1448, "aircraft" : [ {"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1}, {"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9}, {"hex":"a8bcdd","flight":"NKS710 ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0}, {"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2}, {"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1}, {"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9}, {"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2} ] } cat /usr/local/var/dump1090-mut-data/history_75.json { "now" : 1507344034.5, "messages" : 1448, "aircraft" : [ {"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1}, {"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9}, {"hex":"a8bcdd","flight":"NKS710 ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0}, {"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2}, {"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1}, {"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9}, {"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2} ] } There is also an open data API available at https://www.adsbexchange.com/data/# So I grabbed this via REST API: https://public-api.adsbexchange.com/VirtualRadar/AircraftList.json Again using my Latitude and Longitude. Alternative Approach For Ingestion: @Hellmar Becker has a really well developed example and presentation on how he is processing this data. See the Apache NiFi code, Python, Setup Scripts and Presentation here: https://github.com/hellmarbecker/plt-airt-2000 My example is with a different USB stick and a different continent. Resources: http://realadsb.com/ http://realadsb.com/piaware.html https://github.com/mutability/dump1090.git https://www.dzombak.com/blog/2017/01/Monitoring-aircraft-via-ADS-B-on-OS-X.html https://www.faa.gov/nextgen/programs/adsb/ https://community.hortonworks.com/articles/177232/apache-deep-learning-101-processing-apache-mxnet-m.html https://www.dzombak.com/blog/2017/08/Quick-ADS-B-monitoring-on-OS-X.html https://github.com/fredpalmer/flightaware https://walac.github.io/pyusb/ http://www.stuffaboutcode.com/2015/09/read-piaware-flight-data-with-python.html https://github.com/hellmarbecker/plt-airt-2000 https://github.com/jojonas/py1090 https://gist.github.com/fasiha/c123a9c6b6c78df7597bb45e0fed808f ... View more

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time. This is for running Apache MXNet on a Raspberry Pi. Let's get this installed! git clone https://github.com/apache/incubator-mxnet.git The installation instructions at Apache MXNet's website (http://mxnet.incubator.apache.org/install/index.html) are amazing. Pick your platform and your style. I am doing this the simplest way on Linux path. Installation: This builds on previous builds, so see those articles. We installed the drivers for Sense Hat, Intel Movidius and the USB Web Cam previously. Please note that versions for Raspberry Pi, Apache MXNet, Python and other drivers are updated every few months so if you are reading this post DWS 2018 you should check the relevant libraries and update to the latest versions. You need Python, Python Devel and PIP installed and you may need to run as root. You will also need OpenCV installed as mentioned in the previous article. In this combined Python script we grab Sense-Hat sensors for temperature, humidity and more. We also run Movidius image analysis and Apache MXNet Inception on the image that we capture with our web cam. Apache MXNet is now in version 1.1, so you may want to upgrade. pip install --upgrade pip pip install scikit-image git clone https://github.com/tspannhw/mxnet_rpi.git sudo apt-get update -y sudo apt-get install python-pip python-opencv python-scipy python-picamera -y sudo apt-get -y install git cmake build-essential g++-4.8 c++-4.8 liblapack* libblas* libopencv* git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet --branch 1.0.0 cd incubator-mxnet export USE_OPENCV = 0 make cd python pip install --upgrade pip pip install -e . pip install mxnet==1.0.0 MiniFi Flow to Run Python Script and Send Over Images (Running on Raspberry Pi) Routing on Server to Process Either an Image or a JSON Our Apache NiFi Server Receiving Input from Raspberry Pi Apache NiFi Server Processing The Input We route to two different processing flows, with one for saving images, the other adds a schema and converts the JSON data into Apache AVRO. The AVRO content is merged and we send that to a central HDF 3.1 cluster that can write to HDFS. We can either stream to an ACID Hive table or convert AVRO to Apache ORC and store it to HDFS and autogenerate an external Hive table on top of it. You can find many examples of both of these processes in my links below. We could also insert into Apache HBase or insert into an Apache Phoenix table. Or do all of those and send it to Slack, Email, Store in an RDBMS like MySQL and anything else you could think of. Generated Schema Running: We are using Apache MiniFi Java Agent 0.3.0. I will be adding a follow up including MiniFi 0.40 with the native C++ TensorFlow and USB Cam. See this awesome article for TensorFlow: https://community.hortonworks.com/articles/174520/minifi-c-iot-cat-sensor.html Source Code: https://github.com/tspannhw/rpi-mxnet-movidius-minifi This is too easy! References: https://github.com/tspannhw/ApacheBigData101/ https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an-apache-nifi-15-instance-w.html https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an-apache-nifi-15-instance-w.html https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html https://community.hortonworks.com/articles/174399/apache-deep-learning-101-using-apache-mxnet-on-apa.html https://community.hortonworks.com/articles/176784/deep-learning-101-using-apache-mxnet-in-dsx-notebo.html https://community.hortonworks.com/articles/176789/apache-deep-learning-101-using-apache-mxnet-in-apa.html https://community.hortonworks.com/articles/174538/apache-deep-learning-101-using-apache-mxnet-with-h.html https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://community.hortonworks.com/articles/167193/building-and-running-minifi-cpp-in-orangepi-zero.html https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://community.hortonworks.com/articles/155475/powering-apache-minifi-flows-with-a-movidius-neura.html http://mxnet.incubator.apache.org/install/index.html https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html https://github.com/tspannhw/ApacheBigData101 https://github.com/tspannhw/mxnet-in-notebooks https://github.com/tspannhw/nifi-mxnet-yarn/ https://github.com/tspannhw/nvidiajetsontx1-mxnet https://github.com/tspannhw/mxnet_rpi https://github.com/tspannhw/rpi-sensehat-minifi-python/ https://github.com/tspannhw/rpi-minifi-movidius-sensehat ... View more

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time. Another way to work with Apache MXNet is by using your Apache Zeppelin notebook to run your Python deep learning scripts. Apache Zeppelin Notebook As you can see we can format that data as a table using Apache Zeppelin display technology. Use this print statement: print("%table top1pct\ttop1\top2\ttop2pct\ttop3pct\ttop3\ttop4pct\ttop4\ttop5pct\ttop5\timagefilename\truntime\tuuid\n" + top1pct + "\t" + top1 + "\t" + top2pct + "\t" + top2 + "\t" + top3pct + "\t" + top3 + "\t" + top4pct + "\t" + top4 + "\t" + top5pct + "\t" + top5 + "\t" + filename + "\t" + str(round(end - start)) + "\t" + uniqueid + "\n" ) We use the pyspark interpreter to run this Python script, but there's no Spark in here yet. This data also gets loaded in Apache Hive via Apache NiFi as shown here: Deep Learning Models You will need to download the pre-built Inception models and reference them on your server. synset.txt Inception-BN-0000.params Inception-BN-symbol.json See: https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L curl http://data.mxnet.io/models/imagenet/synset.txt More Models http://data.mxnet.io/models/imagenet/ Source Code https://github.com/tspannhw/mxnet-in-notebooks https://github.com/tspannhw/ApacheBigData101 References If you want to run in DSX or Jupyter: https://community.hortonworks.com/articles/176784/deep-learning-101-using-apache-mxnet-in-dsx-notebo.html Setup If you need to setup Apache MXNet on HDF: https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html Other Articles in The Series https://community.hortonworks.com/articles/174538/apache-deep-learning-101-using-apache-mxnet-with-h.html https://community.hortonworks.com/articles/174399/apache-deep-learning-101-using-apache-mxnet-on-apa.html https://community.hortonworks.com/articles/155435/using-the-new-mxnet-model-server.html https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an-apache-nifi-15-instance-w.html ... View more

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time. Many people are using IBM's excellent DSX platform which uses Jupyter Notebooks and the ever popular Kubernetes. I wanted to try out Apache MXNet in this environment. It's great. Create or reuse an existing notebook. For Python, the default is Jupyter. Zeppelin is now also supported. I am using Python 2.7 with DSX Desktop on an OSX workstation. This supports Apache MXNet. My local Apache MXNet installation and MXNet python installation worked well with DSX. I needed OpenCV for this example, so I was able to install right inside IBM DSX via !pip install --user opencv-python. Very easy to start a notebook and add your code, you get nice syntax coloring. I uploaded the precompiled model Here we can check our list of Python libraries with !pip list --isolated --format=columns. Very easy to run your Apache MXNet code right in a notebook. Easy to share with other data scientists and engineers in your group and others. IBM DSX Assets You will need to download the pre-built Inception model and add that to assets. synset.txt Inception-BN-0000.params Inception-BN-symbol.json See: https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L curl http://data.mxnet.io/models/imagenet/synset.txt More Models http://data.mxnet.io/models/imagenet/ Source Code https://github.com/tspannhw/mxnet-in-notebooks https://github.com/tspannhw/ApacheBigData101 ... View more