streaming source

avijeetd — Fri, 16 Sep 2022 10:47:57 GMT

Hi,

I want to create a streaming source for weather data something like the Flume source using TwitterStream api

the weather api is available at https://openweathermap.org/price

Can someone please suggest what's the best way.

Thanks,

Avijeet

Re: streaming source

ldaluz — Wed, 16 Nov 2016 20:08:39 GMT

@Avijeet Dash

I'd suggest using NiFi for this.

You can read from the weather api using NiFi's GetHTTP processor, and use NiFi to process the data before loading it into another system (not sure what other components you're using but NiFi integrates with most systems, so you can use it to directly load the data into HDFS, HBase, Kafka, Cassandra, relational DBs etc..).

Check out the "Fun_with_Hbase" template on the NiFi website to help you get started, it gets random data from an API call before loading into HBase.

https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Also check out this article which uses NiFi to pull data from the Google Finance API:

https://community.hortonworks.com/content/kbentry/8422/visualize-near-real-time-stock-price-changes-using.html

question streaming source in Archives of Support Questions (Read Only)

streaming source

Re: streaming source