New Contributor
Posts: 2
Registered: ‎02-24-2014

Stream HISTORICAL tweets into HDFS

[ Edited ]

Dear Friends,


I have established a setup to stream tweets from Twitter into our HDFS baseed upon the keywords maintained in the FLUME configuration file.


I would like to know if it is possible for FLUME to load the historical/old tweets relevant to those keywords into the HDFS.
If this ain't possible, may I know whether we can achieve this in some other way?



Best Regards,


Posts: 1,903
Kudos: 435
Solutions: 307
Registered: ‎07-31-2013

Re: Stream HISTORICAL tweets into HDFS

There's nothing in Flume that should prevent what you are looking to do. Flume sources feed off of whatever URL or other mechanism you provide it.

If Twitter provides a way for you to stream historical tweets based on a search keyword, provide the same to your Flume source and it will read what is presented to it.
Posts: 21
Registered: ‎02-08-2016

Re: Stream HISTORICAL tweets into HDFS

Hi Ram,


are you able to pull older tweets in hdfs.