Support Questions
Find answers, ask questions, and share your expertise

Flume won't write to hdfs from twitter

Flume won't write to hdfs from twitter

New Contributor

Hello I have been trying to work in a school project and I have been stuck with flume, the Flume gets data from youtube perfectly but it won't store to hdfs. please see the attached image 

Capture.PNG

 

the Flume.conf file is 

 

 

MyTwitAgent.sources = Twitter
MyTwitAgent.channels = MemChannel
MyTwitAgent.sinks = HDFS
MyTwitAgent.sources.Twitter.type = flume.mytwittersource.MyTwitterSourceForFlume
MyTwitAgent.sources.Twitter.channels = MemChannel 
MyTwitAgent.sources.Twitter.consumerKey = XXXXXXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.consumerSecret = XXXXXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.accessToken = XXXXXXXXXXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.accessTokenSecret = XXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.keywords = BIGDATA,DATAANALYTICS,PYTHON,RLANGUAGE
MyTwitAgent.sinks.HDFS.channel = MemChannel
MyTwitAgent.sinks.HDFS.type = hdfs
MyTwitAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8022//user/cloudera/Flume_Tweets
MyTwitAgent.sinks.HDFS.hdfs.fileType = DataStream
MyTwitAgent.sinks.HDFS.hdfs.writeFormat = Text
MyTwitAgent.sinks.HDFS.hdfs.batchSize = 1000
MyTwitAgent.sinks.HDFS.hdfs.rollSize = 0
MyTwitAgent.sinks.HDFS.hdfs.rollCount = 10000
MyTwitAgent.channels.MemChannel.type = memory
MyTwitAgent.channels.MemChannel.capacity = 10000
MyTwitAgent.channels.MemChannel.transactionCapacity = 1000