Created 03-15-2021 03:37 AM
I'm trying to stream data from twitter to hdfs with Flume, i'm using Cloudera Quickstart VM 5.13, i don't have any error but the destination directory is empty.
This is my flume.conf file:
TwitterAgent.channels = MemChannel TwitterAgent.sinks = HDFS TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource TwitterAgent.sources.Twitter.channels = MemChannel TwitterAgent.sources.Twitter.consumerKey = Sp0ti7peTvFPDJSWMGk2ChMZM TwitterAgent.sources.Twitter.consumerSecret = Cncmq5b6rKxWPb6qNSPkqpzIR7L3EcQ8WUCeG0gX4L9sPIzflN TwitterAgent.sources.Twitter.accessToken = 1370386818609377287-IsLuhCt54wK4T2Ua9Cb0TC14rrs1c5 TwitterAgent.sources.Twitter.accessTokenSecret = AL7oYsVUQXz5KXtQSj0tu36R85MyvAsBjcgktdZD63Ou6 TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientist, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing TwitterAgent.sinks.HDFS.channel = MemChannel TwitterAgent.sinks.HDFS.type = hdfs TwitterAgent.sinks.HDFS.hdfs.path = hdfs://quickstart.cloudera:8020/user/flume/tweets/ TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream TwitterAgent.sinks.HDFS.hdfs.writeFormat = text TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000 TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000 TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600 TwitterAgent.channels.MemChannel.type = memory TwitterAgent.channels.MemChannel.capacity = 10000 TwitterAgent.channels.MemChannel.transitionCapacity = 100
I'm invoking this command to stream:
flume-ng agent --conf ./conf/ -f /home/cloudera/flume.conf -n TwitterAgent
Please i want to know on which part i'm doing it wrong. Any valuable suggestion is much appreciated.
Thanks in advance.
Created 03-18-2021 12:22 AM
@Shelton Thank you so much for your help. I can give you the keys to my twitter app so you can test on your end. I created that app only to test Flume, so it's not a big deal if i give you my keys and I will be able to regenerate new ones anyway.
Created 03-18-2021 02:00 AM
Created 03-18-2021 02:29 AM
@Shelton What is your linkedIn?
Created 03-18-2021 02:38 AM
Same name and photo on linkedin profile.
Created 03-18-2021 02:57 AM
@Shelton There are 114 000 results with your name on LinkedIn
Created 03-18-2021 03:05 AM
Created 03-18-2021 03:47 AM
@Shelton I sent you a request.