Support Questions

Find answers, ask questions, and share your expertise

Using Nifi to stream data from Free streaming(not twitter) api to hdfs

Contributor

Actually iam trying to use any api so that i can stream data using nifi and put it to hdfs. I know GetHttp processor is used to fetch data from a website. I want to know any free api thats streams data and also which processor to use in Nifi to connect to that api and stream data from that api using nifi to hdfs. Its a prototype iam trying to build using nifi. I just want to use an api other than twitter. Appreciate any help from friends.

1 ACCEPTED SOLUTION

New Contributor

I am coming in a bit late but want to also recommend randomuser.me

Also want to point out that there are plenty of nifi templates found here: https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Look at the "Fun_with_Hbase" template which gets random data from a rest API and in your case use the PutHDFS processor to ingest in to HDFS.

View solution in original post

4 REPLIES 4

Contributor

Thanks Zlatkin

New Contributor

I am coming in a bit late but want to also recommend randomuser.me

Also want to point out that there are plenty of nifi templates found here: https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Look at the "Fun_with_Hbase" template which gets random data from a rest API and in your case use the PutHDFS processor to ingest in to HDFS.

Contributor

Thanks very much guys for the response. Iam planning to use datasift streaming api and rest api using nifi to store data in hdfs or hive or hbase tables? So i would appreciate your help in choosing the right approach and right nifi processors. Shall i go ahead and use GetHttp and PutHdfs here for both streaming and rest api or shall i think about streaming data directly from nifi to hbase or hive?? please let me know @Juan Sequeiros, @Vladimir Zlatkin thanks guys

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.