Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Super Guru

Ingest All The Things Series: Flight Data Via Radio


I am using the FlightAware Pro Stick Plus ADS-B USB Receiver with Built-in Filter on a Mac, I should hook this up to one of my raspberry pis and add a longer antenna outside. You need a good antenna, a good location and nothing blocking your signal. It also depends on what air traffic is nearby.

For a proof of concept it's pretty cool to see air data going through a cheap USB stick into a computer stored in a file and loaded into Apache NiFi to send on for data processing.

There is a web server you can run to see the planes on a map which is pretty cool, but I want to just ingest the data for processing.

62711-dump1090.png

My Equipment

62761-antenna.png

62762-flightawaredevice.png

If you wish to watch the data flash by in a command-line interface, you can run with interactive flag and watch all the updates.

62710-dumptext1.png

62712-dump1090text.png

62713-dump1090text3.png

62753-dump1090cli.png

62754-dumplive.png

We are dumping the data as it streams as raw text into a file. A snippet of it tailed in Apache NiFi is shown below:

62755-rawlog.png

We are also ingesting ADS-B data that is provided by an open data REST API (https://public-api.adsbexchange.com..) at https://www.adsbexchange.com/.

62756-restdata.png

Like everything else, we may want to add a schema to parse into records.

62757-adsbschemas.png


Our ingest flow:

62758-airtrafficlocalingesthdf.png

I am getting the REST data from the ADSB Exchange REST API, tailing the raw text dump from dump1090 and reading the aircraft history JSON files produced by dump1090 as well.

For further processing, I send the Aircraft History JSON files to my server cluster to send to a cloud hosted MongoDB database. Thanks to a free tier from mLab.

62759-airtrafficserversave.png

And our data quickly arrives as JSON in Mongo.

62760-mongodatareceived.png

The main install is the dump1090 github and is pretty straight forward.


Installation on OSX

brew update

brew install librtlsdr pkg-config

make


Running

./run2.sh >> raw.txt 2>&1


run2.sh
./dump1090 --net --lat 40.265887 --lon -74.534610 --modeac --mlat --write-json-every 1 --json-location-accuracy 2 --write-json /volumes/seagate/projects/dump1090/data

I have entered my local latitude and longitude above. I also write to a local directory that we will read from in Apache NiFi.


Example Data

{ "now" : 1507344034.5,
  "messages" : 1448,
  "aircraft" : [
    {"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1},
    {"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9},
    {"hex":"a8bcdd","flight":"NKS710  ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0},
    {"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2},
    {"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1},
    {"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9},
    {"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2}
  ]
}


cat /usr/local/var/dump1090-mut-data/history_75.json
{ "now" : 1507344034.5,
  "messages" : 1448,
  "aircraft" : [
    {"hex":"a6cb48","lat":40.169403,"lon":-74.526123,"nucp":7,"seen_pos":6.1,"altitude":33000,"mlat":[],"tisb":[],"messages":9,"seen":4.9,"rssi":-6.1},
    {"hex":"a668e2","altitude":17250,"mlat":[],"tisb":[],"messages":31,"seen":4.2,"rssi":-7.9},
    {"hex":"a8bcdd","flight":"NKS710  ","lat":40.205841,"lon":-74.491150,"nucp":7,"seen_pos":1.5,"altitude":9875,"vert_rate":0,"track":45,"speed":369,"category":"A0","mlat":[],"tisb":[],"messages":17,"seen":1.5,"rssi":-5.0},
    {"hex":"a54cd9","mlat":[],"tisb":[],"messages":44,"seen":94.4,"rssi":-7.2},
    {"hex":"a678c3","mlat":[],"tisb":[],"messages":60,"seen":133.2,"rssi":-7.1},
    {"hex":"a1ff83","mlat":[],"tisb":[],"messages":47,"seen":212.3,"rssi":-7.9},
    {"hex":"a24ce0","mlat":[],"tisb":[],"messages":160,"seen":276.3,"rssi":-6.2}
  ]
}


There is also an open data API available at https://www.adsbexchange.com/data/#

So I grabbed this via REST API:

https://public-api.adsbexchange.com/VirtualRadar/AircraftList.json

Again using my Latitude and Longitude.

Alternative Approach For Ingestion:

@Hellmar Becker has a really well developed example and presentation on how he is processing this data. See the Apache NiFi code, Python, Setup Scripts and Presentation here: https://github.com/hellmarbecker/plt-airt-2000 My example is with a different USB stick and a different continent.

Resources:

2,189 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 08:35 AM
Updated by:
 
Contributors
Top Kudoed Authors