Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume agent not able to retrieve tweets based on keywords , it fetch alot of unnecessary data

Flume agent not able to retrieve tweets based on keywords , it fetch alot of unnecessary data

Rising Star

Hello,

 

I see this issue posted at most of platform but nothing works, i tried to used custom jars it didnt work either "Never fetches the class " even you follow flume directory standards. For default twitter source below is my confi file, please help, its been day i am banging my head on this issue. Thanks

 

TwitterAgent.sources = PublicStream
TwitterAgent.channels = MemCh
TwitterAgent.sinks = HDFS

TwitterAgent.sources.PublicStream.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.PublicStream.channels = MemCh
TwitterAgent.sources.PublicStream.consumerKey =
TwitterAgent.sources.PublicStream.consumerSecret =
TwitterAgent.sources.PublicStream.accessToken =-
TwitterAgent.sources.PublicStream.accessTokenSecret =
TwitterAgent.sources.PublicStream.keywords = obama,@realDonaldTrump,#somehashtag

TwitterAgent.sinks.HDFS.channel = MemCh
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://pan0141.panoulu.net:8020/user/hadoop/root/Cork/SocialData/StreamingData/Twitter/KeywordsTweets
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.filePrefix = PublicStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.maxOpenFiles = 10
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10
TwitterAgent.sinks.HDFS.hdfs.rollSize = 100
TwitterAgent.sinks.HDFS.hdfs.rollCount = 100000
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600

TwitterAgent.channels.MemCh.type = memory
TwitterAgent.channels.MemCh.capacity = 10000
TwitterAgent.channels.MemCh.transactionCapacity = 100