Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to fetch tweets using flume and store them in druid


How to fetch tweets using flume and store them in druid

Expert Contributor


I am using hortonworks sandbox 2.6.5. I am trying to fetch tweets from twitter and storing them in druid.

flume.conf file as below:

#Naming the component for current agent 
TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = druidSink

#Configuring the source
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxx
TwitterAgent.sources.Twitter.keywords = Men

#Configuring the sink
agent.sinks.druidSink.type = com.stratio.ingestion.sink.druid.DruidSink
agent.sinks.druidSink.indexService = druid/prod/overlord
agent.sinks.druidSink.discoveryPath = /prod/discovery
agent.sinks.druidSink.dimensions = timestamp,IPAddr,LogLevel,Domain,MessageID

#Configuring the channel 
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

#Binding the source and sink to channel = MemChannel = MemChannel

I have start flume agent using below command:

./bin/flume-ng agent --conf ./conf/ -f conf/flume.conf Dflume.root.logger=DEBUG,console -n TwitterAgent -Xmx3g

How can i check tweets are stored in druid or not ? I am using Druid for the very first time.

Thank you.