Making Data Available

[#1]

Dear community,

I've been wondering something that cannot figure out yet. A quick summary I was able to work for a while with Flume to gather data from twitter and store it in a external table using Hive. After processing data using Hive, i finally got the results that i want. All steps made were manually. What i need now is to make same process, but automatically because i need to finally make it public. A suggestion?

Thanks for your help, Regards

[#2 · Accepted solution]

NiFi is a perfect technology for this: UI/configuration-based, easy connection to twitter stream, automated processing, enterprise security, SDLC and deploy automation ... NiFi usage is growing rapidly and starting to replace flume use cases.

Here are some links on NiFi in general:

• http://hortonworks.com/apache/nifi/
• https://nifi.apache.org/docs.html
• http://www.slideshare.net/hortonworks/design-a-dataflow-in-7-minutes-58718224
• https://community.hortonworks.com/questions/65360/can-someone-help-me-with-apache-nifis-basic-workin...

Here are some How-to articles on NiFi Twitter and NiFi Hive:

• https://community.hortonworks.com/articles/57803/using-nifi-gettwitter-updateattributes-and-replace....
• https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

Here is an article on NiFi SDLC and reusable components

• https://community.hortonworks.com/articles/60868/enterprise-nifi-implementing-reusable-components-a....

[#3]

Hi Greg,

Thank you very much for answering.

Regards