Reply
Highlighted
Contributor
Posts: 40
Registered: ‎06-24-2018

Flume agent not able to retrieve tweets based on keywords , it fetch alot of unnecessary data

Hello,

 

I see this issue posted at most of platform but nothing works, i tried to used custom jars it didnt work either "Never fetches the class " even you follow flume directory standards. For default twitter source below is my confi file, please help, its been day i am banging my head on this issue. Thanks

 

TwitterAgent.sources = PublicStream
TwitterAgent.channels = MemCh
TwitterAgent.sinks = HDFS

TwitterAgent.sources.PublicStream.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.PublicStream.channels = MemCh
TwitterAgent.sources.PublicStream.consumerKey =
TwitterAgent.sources.PublicStream.consumerSecret =
TwitterAgent.sources.PublicStream.accessToken =-
TwitterAgent.sources.PublicStream.accessTokenSecret =
TwitterAgent.sources.PublicStream.keywords = obama,@realDonaldTrump,#somehashtag

TwitterAgent.sinks.HDFS.channel = MemCh
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://pan0141.panoulu.net:8020/user/hadoop/root/Cork/SocialData/StreamingData/Twitter/KeywordsTweets
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.filePrefix = PublicStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.maxOpenFiles = 10
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10
TwitterAgent.sinks.HDFS.hdfs.rollSize = 100
TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600

TwitterAgent.channels.MemCh.type = memory
TwitterAgent.channels.MemCh.capacity = 10000
TwitterAgent.channels.MemCh.transactionCapacity = 100

Announcements
New solutions