streaming source

[#1]

Hi,

I want to create a streaming source for weather data something like the Flume source using TwitterStream api

the weather api is available at https://openweathermap.org/price

Can someone please suggest what's the best way.

Thanks,

Avijeet

[#2 · Accepted solution]

@Avijeet Dash

I'd suggest using NiFi for this.

You can read from the weather api using NiFi's GetHTTP processor, and use NiFi to process the data before loading it into another system (not sure what other components you're using but NiFi integrates with most systems, so you can use it to directly load the data into HDFS, HBase, Kafka, Cassandra, relational DBs etc..).

Check out the "Fun_with_Hbase" template on the NiFi website to help you get started, it gets random data from an API call before loading into HBase.

https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Also check out this article which uses NiFi to pull data from the Google Finance API:

https://community.hortonworks.com/content/kbentry/8422/visualize-near-real-time-stock-price-changes-...