Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume won't write to hdfs from twitter

Flume won't write to hdfs from twitter

New Contributor

Hello I have been trying to work in a school project and I have been stuck with flume, the Flume gets data from youtube perfectly but it won't store to hdfs. please see the attached image 

Capture.PNG

 

the Flume.conf file is 

 

 

MyTwitAgent.sources = Twitter
MyTwitAgent.channels = MemChannel
MyTwitAgent.sinks = HDFS
MyTwitAgent.sources.Twitter.type = flume.mytwittersource.MyTwitterSourceForFlume
MyTwitAgent.sources.Twitter.channels = MemChannel 
MyTwitAgent.sources.Twitter.consumerKey = XXXXXXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.consumerSecret = XXXXXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.accessToken = XXXXXXXXXXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.accessTokenSecret = XXXXXXXXXXXXXXXXXX
MyTwitAgent.sources.Twitter.keywords = BIGDATA,DATAANALYTICS,PYTHON,RLANGUAGE
MyTwitAgent.sinks.HDFS.channel = MemChannel
MyTwitAgent.sinks.HDFS.type = hdfs
MyTwitAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8022//user/cloudera/Flume_Tweets
MyTwitAgent.sinks.HDFS.hdfs.fileType = DataStream
MyTwitAgent.sinks.HDFS.hdfs.writeFormat = Text
MyTwitAgent.sinks.HDFS.hdfs.batchSize = 1000
MyTwitAgent.sinks.HDFS.hdfs.rollSize = 0
MyTwitAgent.sinks.HDFS.hdfs.rollCount = 10000
MyTwitAgent.channels.MemChannel.type = memory
MyTwitAgent.channels.MemChannel.capacity = 10000
MyTwitAgent.channels.MemChannel.transactionCapacity = 1000

 

Don't have an account?
Coming from Hortonworks? Activate your account here