Ingest All The Things Series: Flight Data Via Radio
I am using the FlightAware Pro Stick Plus ADS-B USB Receiver with Built-in Filter on a Mac, I should hook this up to one of my raspberry pis and add a longer antenna outside. You need a good antenna, a good location and nothing blocking your signal. It also depends on what air traffic is nearby.
For a proof of concept it's pretty cool to see air data going through a cheap USB stick into a computer stored in a file and loaded into Apache NiFi to send on for data processing.
There is a web server you can run to see the planes on a map which is pretty cool, but I want to just ingest the data for processing.
My Equipment
If you wish to watch the data flash by in a command-line interface, you can run with interactive flag and watch all the updates.
We are dumping the data as it streams as raw text into a file. A snippet of it tailed in Apache NiFi is shown below:
We are also ingesting ADS-B data that is provided by an open data REST API (https://public-api.adsbexchange.com..) at https://www.adsbexchange.com/.
Like everything else, we may want to add a schema to parse into records.
Our ingest flow:
I am getting the REST data from the ADSB Exchange REST API, tailing the raw text dump from dump1090 and reading the aircraft history JSON files produced by dump1090 as well.
For further processing, I send the Aircraft History JSON files to my server cluster to send to a cloud hosted MongoDB database. Thanks to a free tier from mLab.
And our data quickly arrives as JSON in Mongo.
The main install is the dump1090 github and is pretty straight forward.
Installation on OSX
brew update brew install librtlsdr pkg-config make
Running
./run2.sh >> raw.txt 2>&1 run2.sh ./dump1090 --net --lat 40.265887 --lon -74.534610 --modeac --mlat --write-json-every 1 --json-location-accuracy 2 --write-json /volumes/seagate/projects/dump1090/data
I have entered my local latitude and longitude above. I also write to a local directory that we will read from in Apache NiFi.
Example Data
{ "now" : 1507344034.5,
"messages" : 1448,
"aircraft" : [
{"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1},
{"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9},
{"hex":"a8bcdd","flight":"NKS710 ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0},
{"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2},
{"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1},
{"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9},
{"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2}
]
}
cat /usr/local/var/dump1090-mut-data/history_75.json
{ "now" : 1507344034.5,
"messages" : 1448,
"aircraft" : [
{"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1},
{"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9},
{"hex":"a8bcdd","flight":"NKS710 ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0},
{"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2},
{"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1},
{"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9},
{"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2}
]
}
There is also an open data API available at https://www.adsbexchange.com/data/#
So I grabbed this via REST API:
https://public-api.adsbexchange.com/VirtualRadar/AircraftList.json
Again using my Latitude and Longitude.
Alternative Approach For Ingestion:
@Hellmar Becker has a really well developed example and presentation on how he is processing this data. See the Apache NiFi code, Python, Setup Scripts and Presentation here: https://github.com/hellmarbecker/plt-airt-2000 My example is with a different USB stick and a different continent.
Resources:
- http://realadsb.com/
- http://realadsb.com/piaware.html
- https://github.com/mutability/dump1090.git
- https://www.dzombak.com/blog/2017/01/Monitoring-aircraft-via-ADS-B-on-OS-X.html
- https://www.faa.gov/nextgen/programs/adsb/
- https://community.hortonworks.com/articles/177232/apache-deep-learning-101-processing-apache-mxnet-m...
- https://www.dzombak.com/blog/2017/08/Quick-ADS-B-monitoring-on-OS-X.html
- https://github.com/fredpalmer/flightaware
- https://walac.github.io/pyusb/
- http://www.stuffaboutcode.com/2015/09/read-piaware-flight-data-with-python.html
- https://github.com/hellmarbecker/plt-airt-2000
- https://github.com/jojonas/py1090
- https://gist.github.com/fasiha/c123a9c6b6c78df7597bb45e0fed808f
