Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Twitter flume ingestion

Twitter flume ingestion

New Contributor

## Config file 

 

# Initialize agent's source, channel and sink
twitteragent.sources = twitter
twitteragent.channels = memorychannel
twitteragent.sinks = HDFS

# Setting the source to spool directory where the file exists
twitteragent.sources.twitter.type = org.apache.flume.source.twitter.TwitterSource
twitteragent.sources.twitter.channels= memorychannel

#keys and token of sources

twitteragent.sources.twitter.consumerkey= ######################################
twitteragent.sources.twitter.consumersecret = ##################################
twitteragent.sources.twitter.accesstoken = ##########################################
twitteragent.sources.twitter.accesstokensecret = ###################################

twitteragent.sources.twitter.keywords= squash,docker,data scientist

# Setting the channel to memory
twitteragent.channels.memorychannel.type = memory

# Max number of events stored in the memory channel
twitteragent.channels.memorychannel.capacity = 100000
twitteragent.channels.memorychannel.transactioncapacity = 10000

# Setting the sink to HDFS

twitteragent.sinks.HDFS.type = hdfs
twitteragent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/tmp/twitteringestion
twitteragent.sinks.HDFS.hdfs.fileType = DataStream

# Write format can be text or writable
twitteragent.sinks.HDFS.hdfs.writeFormat = Text

# rollover file based on maximum size of 10 MB
twitteragent.sinks.HDFS.hdfs.rollSize = 10485760
twiiteragent.sinks.HDFS.hdfs.batchsize =1000

# Connect source and sink with channel
twitteragent.sources.twitter.channels = memorychannel
twitteragent.sinks.HDFS.channel = memorychannel

 

 

 

### ruuning then 
[cloudera@localhost flume-ng]$ bin/flume-ng agent --conf ./conf/ -f conf/flume1-conf.properties -Dflume.root.logger=DEBUG,console -n twitteragent

 

 

### getting error 

2016-08-26 02:55:02,672 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:523)] Hadoop Security enabled: false
2016-08-26 02:55:02,678 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel memorychannel connected to [HDFS]
2016-08-26 02:55:02,699 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@22a98932 counterGroup:{ name:null counters:{} } }} channels:{memorychannel=org.apache.flume.channel.MemoryChannel{name: memorychannel}} }
2016-08-26 02:55:02,726 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel memorychannel
2016-08-26 02:55:02,729 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:160)] Waiting for channel: memorychannel to start. Sleeping for 500 ms
2016-08-26 02:55:02,848 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: memorychannel: Successfully registered new MBean.
2016-08-26 02:55:02,848 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: memorychannel started
2016-08-26 02:55:03,229 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink HDFS
2016-08-26 02:55:03,237 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SINK, name: HDFS: Successfully registered new MBean.
2016-08-26 02:55:03,237 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SINK, name: HDFS started
2016-08-26 02:55:03,239 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2016-08-26 02:55:33,232 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:56:03,233 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:56:33,234 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:57:03,235 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:57:33,236 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:58:03,237 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:58:33,237 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:59:03,239 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:59:33,240 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:00:03,241 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:00:33,242 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:01:03,243 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:01:33,244 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:02:03,245 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:02:33,246 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:03:03,247 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:03:33,248 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:04:03,249 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:04:33,251 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:05:03,253 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:05:33,254 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:06:03,255 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:06:33,256 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:07:03,257 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:07:33,258 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:08:03,259 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:08:33,260 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:09:03,261 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:09:33,262 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:10:03,264 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:10:33,265 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:11:03,266 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:11:33,267 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:12:03,269 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:12:33,270 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:13:03,271 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:13:33,272 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:14:03,273 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:14:33,275 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:15:03,276 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:15:33,277 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:16:03,279 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes

 

 

please help me on this asap.

Thanks

2 REPLIES 2

Re: Twitter flume ingestion

Super Collaborator
It doesn't appear there are any errors in this log. What is the error you are observing?

-pd
Highlighted

Re: Twitter flume ingestion

New Contributor

Actually this is keep repeating and nothing is getting inserted , the process goes on for more than 20 -25 min but nothing happens .

 

I am not sure if this is error or not

 

Component type: SINK, name: HDFS started
2016-08-26 02:55:03,239 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2016-08-26 02:55:33,232 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes

 

how should i proceed furthure ?