Where in the world is Tim Spann, The NiFi Guy?
I recommend the BU-353-S4 USB GPS, it works well with Raspberry PIs and is very affordable. Connecting this to a RPIWZ, I can run this on a small battery and bring this everywhere for location tracking. Put it on your delivery truck, chasis, train, plane, drone, robot and more. I'll track myself so I can own the data.
What do these GPS Fields Mean?
EPS - Error Estimate in Meter/Second
EPX = Estimated Longitude Error in Meters
EPV = Estimated Vertical Error in Meters
EPT = Estimated Timestamp Error
Speed = Speed !!!
Climb = Climb (Positive) or Sink (Negative) rate in meters per second of upwards or downwards movement.
Track = Course over ground in degrees from True North
Mode = NMEA mode; values are 0 - NA, 1 - No Fix, 2D and 3D.
My question is if you already have an estimate of the error, do something about it!!!
Python Walk Through
First install the utilities you need for GPS and Python. We also install NTP to get as accurate time as possible.
sudo apt-get install gpsd gpsd-clients python-gps ntp
For testing to make sure that everything works, try two of these GPS utilities. Make sure you have the USB plugged in, you will need a RPIZero adapter to convert from little USB to normal size. I then connect a small USB hub to connect the GPS unit as well as sometimes mouse and keyboard. Get one of these, you will need it. You will also need an adapter from little to full size HDMI. You only really need the mouse, keyboard and monitor while you are doing the first setup up WIFI. Once that's setup, just SSH into your device and forget it.
cgps
gpsmon
gpxlogger
dumps XML data in GPX format
gpspipe -l -o test.json -p -w -n 10
without -o goes to STDOUT/STIN
These will work from the command line and give you a read out. It will take a few seconds or maybe a minute the first time to calibrate. If you don't get any numbers, stick your GPS on a window or put it outside.
If you have to manually run the GPS demon:
gpsd -n -D 2 /dev/ttyUSB0
I found some code to read the GPS sensor over USB with Python. From there I modified the code for a slower refresh and no UI as I want to just send this data to Apache NiFi over MQTT using Eclipse Paho MQTT client.
One enhancement I have considered is an offline mode to save all the data as a buffer and then on reconnect mass send the rest. Also you could search for other WiFi signals and try to use open and free ones. You probably want to then add SQL, encryption and some other controls. Or you could install and use Apache MiniFi Java or C++ agent on the Zero.
#! /usr/bin/python # Based on # Written by Dan Mandle http://dan.mandle.me September 2012 # License: GPL 2.0 import os from gps import * from time import * import time import threading import json import paho.mqtt.client as paho gpsd = None class GpsPoller(threading.Thread): def __init__(self): threading.Thread.__init__(self) global gpsd #bring it in scope gpsd = gps(mode=WATCH_ENABLE) #starting the stream of info self.current_value = None self.running = True #setting the thread running to true def run(self): global gpsd while gpsp.running: gpsd.next() #this will continue to loop and grab EACH set of gpsd info to clear the buffer if __name__ == '__main__': gpsp = GpsPoller() # create the thread try: gpsp.start() # start it up while True: if gpsd.fix.latitude > 0: row = [ { 'latitude': str(gpsd.fix.latitude), 'longitude': str(gpsd.fix.longitude), 'utc': str(gpsd.utc), 'time': str(gpsd.fix.time), 'altitude': str(gpsd.fix.altitude), 'eps': str(gpsd.fix.eps), 'epx': str(gpsd.fix.epx), 'epv': str(gpsd.fix.epv), 'ept': str(gpsd.fix.ept), 'speed': str(gpsd.fix.speed), 'climb': str(gpsd.fix.climb), 'track': str(gpsd.fix.track), 'mode': str(gpsd.fix.mode)} ] json_string = json.dumps(row) client = paho.Client() client.username_pw_set("jrfcwrim","UhBGemEoqf0D") client.connect("m13.cloudmqtt.com", 14162, 60) client.publish("rpiwzgps", payload=json_string, qos=0, retain=True) time.sleep(60) except (KeyboardInterrupt, SystemExit): #when you press ctrl+c gpsp.running = False gpsp.join() # wait for the thread to finish what it's doing
Example JSON Data
{"track": "0.0", "speed": "0.0", "utc": "2017-05-01T23:49:46.000Z", "epx": "8.938", "epv": "29.794", "altitude": "40.742", "eps": "23.66", "longitude": "-74.529216408", "mode": "3", "time": "2017-05-01T23:49:46.000Z", "latitude": "40.268141521", "climb": "0.0", "ept": "0.005"}Ingest Any Data Any Where Any Time Any Dimension in Time and Space
1. ConsumeMQTT
2. InferAvroSchema
3. ConvertJSONtoAvro
4. MergeContent
5. ConvertAvroToORC
6. PutHDFS
Visualize Whirled Peas
We turn this raw data into Hive tables and then visualize into pretty tables and charts with Apache Zeppelin. You could also report with any ODBC and JDBC reporting tool like Tableau or PowerBI.
Store It Where?
su hdfs hdfs dfs -mkdir -p /rpwz/gps hdfs dfs -chmod -R 777 /rpwz/gps
If you store it you can query it!
Why Do I love Apache NiFi? Let me count the ways....
Instead of hand-rolling some Hive DDL, NiFi will automagically generate all the DDL I need based on an inferred AVRO Schema (soon using Schema Registry lookup!!!). So you can easily drop in a file, convert to ORC, save to HDFS, generate an external Hive table and query it in seconds. All with no coding. Very easy to wire this to send a message to a front end via Web Sockets, JMS, AJAX, ... So we can drop a file in S3 or HDFS, convert to ORC for mega fast LLAP queries and tell a front-end what the table is and it could query it.
References:
- http://www.danmandle.com/blog/getting-gpsd-to-work-with-python/
- http://www.d.umn.edu/~and02868/projects.html
- http://kingtidesailing.blogspot.com/2016/02/connect-globalsat-bu-353-usb-gps-to.html
- http://blog.petrilopia.net/linux/raspberry-pi-gps-working/
- https://community.hortonworks.com/articles/72420/ingesting-remote-sensor-feeds-into-apache-phoenix.h...
- https://www.systutorials.com/docs/linux/man/1-gpspipe/
- https://www.systutorials.com/docs/linux/man/1-gps/
- http://usglobalsat.com/p-688-bu-353-s4.aspx https://www.raspberrypi.org/products/pi-zero-w/
Source Code:
Quick Tip:
Utilize Apache NiFi's scheduler to limit the # of calls you make to third-party services, a single NiFi instance can easily overwhelm most free tiers of services. I made 122 calls to Weather Underground in a few seconds. So set those times! for weather once every 15 minutes or 30 or even 1 hour is good.
Note:
GPS information can also be read from drones, cars, phones and lots of custom sensors in IIOT devices.
