Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

streaming source

Solved Go to solution

streaming source

Expert Contributor

Hi,

I want to create a streaming source for weather data something like the Flume source using TwitterStream api

the weather api is available at https://openweathermap.org/price

Can someone please suggest what's the best way.

Thanks,

Avijeet

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: streaming source

Rising Star

@Avijeet Dash

I'd suggest using NiFi for this.

You can read from the weather api using NiFi's GetHTTP processor, and use NiFi to process the data before loading it into another system (not sure what other components you're using but NiFi integrates with most systems, so you can use it to directly load the data into HDFS, HBase, Kafka, Cassandra, relational DBs etc..).

Check out the "Fun_with_Hbase" template on the NiFi website to help you get started, it gets random data from an API call before loading into HBase.

https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Also check out this article which uses NiFi to pull data from the Google Finance API:

https://community.hortonworks.com/content/kbentry/8422/visualize-near-real-time-stock-price-changes-...

1 REPLY 1
Highlighted

Re: streaming source

Rising Star

@Avijeet Dash

I'd suggest using NiFi for this.

You can read from the weather api using NiFi's GetHTTP processor, and use NiFi to process the data before loading it into another system (not sure what other components you're using but NiFi integrates with most systems, so you can use it to directly load the data into HDFS, HBase, Kafka, Cassandra, relational DBs etc..).

Check out the "Fun_with_Hbase" template on the NiFi website to help you get started, it gets random data from an API call before loading into HBase.

https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Also check out this article which uses NiFi to pull data from the Google Finance API:

https://community.hortonworks.com/content/kbentry/8422/visualize-near-real-time-stock-price-changes-...

Don't have an account?
Coming from Hortonworks? Activate your account here