Cloudera Community
TimothySpann
user rank
user rank
Master Guru

Ingesting iBeacon Data via BLE to MQTT WiFi Gatewa...

by Master Guru Master Guru in Community Articles
‎04-28-2017 09:39 PM
3 Kudos
‎04-28-2017 09:39 PM
3 Kudos
People were asking me when you could throw some more acronyms into an article. So I did that. iBeacons are tiny devices that are broadcasting over Bluetooth Low Energy (BLE) that are very good for advertising and for broadcasting proximity. I have been thinking of the retail space so I purchased a beacon, got a beacon simulator on my phone and found a really cool gateway device. It reads the BLE transmissions from these devices, packages them as JSON MQTT messages and sends them over WIFI to a MQTT Broker of your choice. I am using a cloud hosted MQTT broker to hold these and then Apache NiFi subscribes to these messages. This can happen amazingly fast as MQTT is very light weight, NIFI is fast and BLE is fast. These tiny messages are constantly advertising their identity and other information. With multiple beacons in a building, store, room, facility; you can see who is close to where. As you might expect there's multiple standards for this, iBeacon is Apple's and Google Eddystone. Fortunately this gateway works with both and so can you. Walk into the Physical Web. Use Cases Happy Bubbles Technology In Apache NiFi, it doesn't matter the data or location, it's easy as pie to ingest Beacon data via MQTT and store it in ORC files for Hive table queries. Easy to visualize and analyze the data in Apache Zeppelin against these Hive tables. What I like about this gateway from Happy Bubble technology besides the awesome Hippo logo is that this gateway is dead simple to setup. Plug it in, wait a bit, connect to it's WIFI, set WIFI and MQTT settings. Then you are running! They also provide an open source server you can use to see the data. If BLE messages are ibeacon format, they will get one JSON format. If they are Google Eddystone format it's a second one. And if it's something else, it will get a generic format. I have three listeners in NiFi to grab them all. Example Messages happy-bubbles-ble {"rssi":-63, "heap_free":27912} {"hostname": "happy-bubbles-ble", "mac": "fa909c522836", "rssi": -51, "is_scan_response": "1", "type": "0", "data": "020a000816f0ff640000000011094d696e69426561636f6e5f3331303033"} {"hostname": "happy-bubbles-ble", "beacon_type": "ibeacon", "mac": "fa909c522836", "rssi": -51, "uuid": "e2c56db5dffb48d2b060d0f5a71096e0", "major": "0000", "minor": "0000", "tx_power": "c5"} Phones There's a great free application for IPhone called Locate Beacon that let's you simulate beacons. This is great for testing. For the inexpensive beacon from Happy Bubbles I bought it is a minew MS49_nrf51822 and there's an app for that called BeaconSET. This will let you set properties and see it over your phone's BlueTooth. DDL CREATE EXTERNAL TABLE IF NOT EXISTS beaconstatus (rssi INT, heap_free INT) STORED AS ORC LOCATION '/beacons'; CREATE EXTERNAL TABLE IF NOT EXISTS ibeacon (hostname STRING, beacon_type STRING, mac STRING, rssi INT, uuid STRING, major STRING, minor STRING, tx_power STRING) STORED AS ORC LOCATION '/beacons/ibeacon'; CREATE EXTERNAL TABLE IF NOT EXISTS beacongateway (hostname STRING, mac STRING, rssi INT, is_scan_response STRING, type STRING, data STRING) STORED AS ORC LOCATION '/beacons/gateway'; Reference https://en.wikipedia.org/wiki/IBeacon https://en.wikipedia.org/wiki/Bluetooth_low_energy http://developer.estimote.com/ibeacon/tutorial/part-3-ranging-beacons/ https://en.wikipedia.org/wiki/MQTT https://developers.google.com/beacons/proximity/guides https://www.beaconzone.co.uk/BeaconTriggerDataAndServers https://developer.apple.com/ibeacon/ https://developers.google.com/beacons/ https://www.happybubbles.tech/presence/detector https://www.happybubbles.tech/presence/docs/setup/ https://github.com/happy-bubbles/ https://itunes.apple.com/us/app/locate-beacon/id738709014?mt=8 ... View more
Labels:

Re: Store a Flow to Disk and Then Reserialize It t...

by Master Guru Master Guru in Community Articles
‎04-28-2017 02:58 AM
‎04-28-2017 02:58 AM
i have not tried, but there's no reason you couldn't. and you could send it to a remote cluster via site 2 site ... View more

Re: Nifi for batch ingest

by Master Guru Master Guru in Support Questions
‎04-24-2017 04:49 PM
‎04-24-2017 04:49 PM
Define large file? Do you mean megs? gigs? terabytes? petabytes? For 100+ megs, NIFI blasts through that. Gigs are fine too. If the terabytes are streaming in that's fine too. Once you get huge, you will need many cores, much RAM and many nodes. It will take similar horsepower that was required to do the same processing in Hadoop nodes. So 256GIG nodes with 32 cores and a few dozen will handle almost anything. https://nifi.apache.org/docs.html You will be limited by disk speed (~50m) and network. Network saturation could be a worry first. get 10GB/s+ networks. SSD or faster drives. ... View more

Re: NiFi processor to fetch attribute value dynami...

by Master Guru Master Guru in Support Questions
‎04-19-2017 07:47 PM
‎04-19-2017 07:47 PM
cool, put a JIRA ticket out for that and post it to your own github. perhaps that will get mainlined. Accept your answer. ... View more

Re: NiFi processor to fetch attribute value dynami...

by Master Guru Master Guru in Support Questions
‎04-19-2017 04:31 PM
1 Kudo
‎04-19-2017 04:31 PM
1 Kudo
Scan attribute should work just put in a regular expression that does what you need and just use the one attribute for . maybe it's updated in the new one. You can create your own processor, starting with the code in ScanAttribute if it doesn't meet your needs then deploy your NAR to the nifi/lib dir and restart nifi. This is the formatting https://github.com/tspannhw/nifi-nlp-processor copy this code https://github.com/apache/nifi/blob/f8cad0f8c94869e2fd2a30896033c3c48eadbf14/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ScanAttribute.java https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestScanAttribute.java ... View more

Re: Dose HDP support IBM Power Systems Servers?

by Master Guru Master Guru in Support Questions
‎04-19-2017 12:05 PM
‎04-19-2017 12:05 PM
Mid 2017! Yes ... View more

Re: Dose HDP support IBM Power Systems Servers?

by Master Guru Master Guru in Support Questions
‎04-19-2017 12:05 PM
‎04-19-2017 12:05 PM
https://www.ibm.com/developerworks/library/l-openstack-cloud-on-ibm-power-systems-trs/ ... View more

Re: Druid Superset Wont Start on HDP 2.6

by Master Guru Master Guru in Support Questions
‎04-14-2017 11:23 PM
‎04-14-2017 11:23 PM
That worked! ... View more

Re: Druid Superset Wont Start on HDP 2.6

by Master Guru Master Guru in Support Questions
‎04-14-2017 11:21 PM
‎04-14-2017 11:21 PM
setting to mysql. i actually created the mysql database for it, but didn't switch to mysql. trying now. ... View more

Druid Superset Wont Start on HDP 2.6

by Master Guru Master Guru in Support Questions
‎04-12-2017 09:08 PM
1 Kudo
‎04-12-2017 09:08 PM
1 Kudo
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/DRUID/0.9.2/package/scripts/superset.py", line 169, in <module> Superset().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/DRUID/0.9.2/package/scripts/superset.py", line 108, in start self.configure(env, upgrade_type=upgrade_type) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 117, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/DRUID/0.9.2/package/scripts/superset.py", line 84, in configure user=params.druid_user) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'source /etc/superset/conf/superset-env.sh ; /usr/hdp/current/druid-superset/bin/superset db upgrade' returned 1. /usr/hdp/2.6.0.3-8/superset/lib/python3.4/importlib/_bootstrap.py:1161: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead. spec.loader.load_module(spec.name) /usr/hdp/2.6.0.3-8/superset/lib/python3.4/importlib/_bootstrap.py:1161: ExtDeprecationWarning: Importing flask.ext.script is deprecated, use flask_script instead. spec.loader.load_module(spec.name) Loaded your LOCAL configuration Traceback (most recent call last): File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 337, in dbapi from sqlite3 import dbapi2 as sqlite # try 2.5+ stdlib name. File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/sqlite3/__init__.py", line 23, in <module> from sqlite3.dbapi2 import * File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/sqlite3/dbapi2.py", line 27, in <module> from _sqlite3 import * ImportError: No module named '_sqlite3' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/hdp/current/druid-superset/bin/superset", line 84, in <module> from superset.cli import manager File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/superset/__init__.py", line 36, in <module> utils.pessimistic_connection_handling(db.engine.pool) File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/flask_sqlalchemy/__init__.py", line 816, in engine return self.get_engine(self.get_app()) File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/flask_sqlalchemy/__init__.py", line 833, in get_engine return connector.get_engine() File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/flask_sqlalchemy/__init__.py", line 499, in get_engine self._engine = rv = sqlalchemy.create_engine(info, **options) File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/sqlalchemy/engine/__init__.py", line 387, in create_engine return strategy.create(*args, **kwargs) File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 80, in create dbapi = dialect_cls.dbapi(**dbapi_args) File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 339, in dbapi raise e File "/usr/hdp/2.6.0.3-8/superset/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 334, in dbapi from pysqlite2 import dbapi2 as sqlite ImportError: No module named 'pysqlite2' ... View more
Contact Me
Online
Offline
Last Visited
‎05-20-2024 05:42 PM
Community Statistics
Member Since ‎01-07-2019 11:58 AM
Last Visited ‎05-20-2024 05:42 PM
Posts 1,973
Kudos received 1122