Support Questions

Find answers, ask questions, and share your expertise

Not able to stream twitter data in to hdfs with flume

avatar
Explorer

I'm trying to stream data from twitter to hdfs with Flume, i'm using Cloudera Quickstart VM 5.13, i don't have any error but the destination directory is empty.

This is my flume.conf file:

TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = Sp0ti7peTvFPDJSWMGk2ChMZM
TwitterAgent.sources.Twitter.consumerSecret = Cncmq5b6rKxWPb6qNSPkqpzIR7L3EcQ8WUCeG0gX4L9sPIzflN
TwitterAgent.sources.Twitter.accessToken = 1370386818609377287-IsLuhCt54wK4T2Ua9Cb0TC14rrs1c5
TwitterAgent.sources.Twitter.accessTokenSecret = AL7oYsVUQXz5KXtQSj0tu36R85MyvAsBjcgktdZD63Ou6
TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientist, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://quickstart.cloudera:8020/user/flume/tweets/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = text 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transitionCapacity = 100

I'm invoking this command to stream:

flume-ng agent --conf ./conf/ -f /home/cloudera/flume.conf -n TwitterAgent

Please i want to know on which part i'm doing it wrong. Any valuable suggestion is much appreciated.

Thanks in advance.

16 REPLIES 16

avatar
Explorer

@Shelton Thank you so much for your help. I can give you the keys to my twitter app so you can test on your end. I created that app only to test Flume, so it's not a big deal if i give you my keys and I will be able to regenerate new ones anyway.

avatar
Master Mentor

@emeric 

Cool so how will you send it safely?  check my linkedin should be easy to connect 🙂 

 

avatar
Explorer

@Shelton What is your linkedIn? 

avatar
Master Mentor

@emeric 

Same name and photo on linkedin profile. 

avatar
Explorer

@Shelton There are 114 000 results with your name on LinkedIn

avatar
Master Mentor

avatar
Explorer

@Shelton I sent you a request.