Support Questions
Find answers, ask questions, and share your expertise

How to fetch tweets using flume and store them in druid

How to fetch tweets using flume and store them in druid

Expert Contributor


I am using hortonworks sandbox 2.6.5. I am trying to fetch tweets from twitter and storing them in druid.

flume.conf file as below:

#Naming the component for current agent 
TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = druidSink

#Configuring the source
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxx
TwitterAgent.sources.Twitter.keywords = Men

#Configuring the sink
agent.sinks.druidSink.type = com.stratio.ingestion.sink.druid.DruidSink
agent.sinks.druidSink.indexService = druid/prod/overlord
agent.sinks.druidSink.discoveryPath = /prod/discovery
agent.sinks.druidSink.dimensions = timestamp,IPAddr,LogLevel,Domain,MessageID

#Configuring the channel 
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

#Binding the source and sink to channel = MemChannel = MemChannel

I have start flume agent using below command:

./bin/flume-ng agent --conf ./conf/ -f conf/flume.conf Dflume.root.logger=DEBUG,console -n TwitterAgent -Xmx3g

How can i check tweets are stored in druid or not ? I am using Druid for the very first time.

Thank you.

Don't have an account?