Real time data from sensors

[#1]

Hi,

I am new to nifi and i would like to ask if someone has experience to integrate real time sensor data into Nifi.

My idea is to get data from: https://www.pubnub.com/developers/realtime-data-streams/sensor-network/

Then store into Hadoop and visualize it,

Any ideas? What processors I can use?

Thanks!

[#2 · Accepted solution]

@jordi

There are a few ways to do this within NiFi, but I would recommend using the ExecuteScript processor. You can use this processor to execute a python script (which is in the link you provided).

The response from pubnub can be captured using python (as json) and the routed to the EvaluateJsonPath processor, where the json variables can be parsed and processed as part of your NiFi flow.

NiFi can also route the json or parsed values to HDFS, HBase, Hive, etc using the corresponding processors.

Let me know if this helps.

[#3]

Hi Dan, thanks for your quick answer. I will check.

Thanks!

[#4]

Hi, I have another question if i want to send this realtime data for visualization in real time, how can I connect Nifi to Grafana chart for example? Should I use kafka or do you recommed others?

Thanks!

[#5]

@jordi

You can use the PostHTTP processor to send json (or another format) to Grafana using their API. Depending on your use case, you may want to add Kafka in there as the intermediate message/routing bus, especially if you are sending that data to other endpoints/services in addition to Grafana.

An alternative option would be to send the data into Solr (using the PutSolrContentStream NiFi processor). From Solr, you can use the Banana UI to create realtime dashboard.

If this answers your question, can you please click the "Accept" response. 🙂 And if you have additional questions or would like more detail, please let me know.