question Real time data from sensors in Archives of Support Questions (Read Only)

Real time data from sensors

jamatller — Fri, 16 Sep 2022 10:45:06 GMT

Hi,

I am new to nifi and i would like to ask if someone has experience to integrate real time sensor data into Nifi.

My idea is to get data from: https://www.pubnub.com/developers/realtime-data-streams/sensor-network/

Then store into Hadoop and visualize it,

Any ideas? What processors I can use?

Thanks!

Re: Real time data from sensors

dzaratsian — Fri, 21 Oct 2016 08:50:08 GMT

@jordi

There are a few ways to do this within NiFi, but I would recommend using the ExecuteScript processor. You can use this processor to execute a python script (which is in the link you provided).

The response from pubnub can be captured using python (as json) and the routed to the EvaluateJsonPath processor, where the json variables can be parsed and processed as part of your NiFi flow.

NiFi can also route the json or parsed values to HDFS, HBase, Hive, etc using the corresponding processors.

Let me know if this helps.

Re: Real time data from sensors

jamatller — Fri, 21 Oct 2016 16:10:18 GMT

Hi Dan, thanks for your quick answer. I will check.

Thanks!

Re: Real time data from sensors

jamatller — Fri, 21 Oct 2016 18:41:37 GMT

Hi, I have another question if i want to send this realtime data for visualization in real time, how can I connect Nifi to Grafana chart for example? Should I use kafka or do you recommed others?

Thanks!

Re: Real time data from sensors

dzaratsian — Fri, 21 Oct 2016 19:57:10 GMT

@jordi

You can use the PostHTTP processor to send json (or another format) to Grafana using their API. Depending on your use case, you may want to add Kafka in there as the intermediate message/routing bus, especially if you are sending that data to other endpoints/services in addition to Grafana.

An alternative option would be to send the data into Solr (using the PutSolrContentStream NiFi processor). From Solr, you can use the Banana UI to create realtime dashboard.

If this answers your question, can you please click the "Accept" response. 🙂 And if you have additional questions or would like more detail, please let me know.