Reply
New Contributor
Posts: 3
Registered: ‎08-26-2016

Twitter flume ingestion

## Config file 

 

# Initialize agent's source, channel and sink
twitteragent.sources = twitter
twitteragent.channels = memorychannel
twitteragent.sinks = HDFS

# Setting the source to spool directory where the file exists
twitteragent.sources.twitter.type = org.apache.flume.source.twitter.TwitterSource
twitteragent.sources.twitter.channels= memorychannel

#keys and token of sources

twitteragent.sources.twitter.consumerkey= ######################################
twitteragent.sources.twitter.consumersecret = ##################################
twitteragent.sources.twitter.accesstoken = ##########################################
twitteragent.sources.twitter.accesstokensecret = ###################################

twitteragent.sources.twitter.keywords= squash,docker,data scientist

# Setting the channel to memory
twitteragent.channels.memorychannel.type = memory

# Max number of events stored in the memory channel
twitteragent.channels.memorychannel.capacity = 100000
twitteragent.channels.memorychannel.transactioncapacity = 10000

# Setting the sink to HDFS

twitteragent.sinks.HDFS.type = hdfs
twitteragent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/tmp/twitteringestion
twitteragent.sinks.HDFS.hdfs.fileType = DataStream

# Write format can be text or writable
twitteragent.sinks.HDFS.hdfs.writeFormat = Text

# rollover file based on maximum size of 10 MB
twitteragent.sinks.HDFS.hdfs.rollSize = 10485760
twiiteragent.sinks.HDFS.hdfs.batchsize =1000

# Connect source and sink with channel
twitteragent.sources.twitter.channels = memorychannel
twitteragent.sinks.HDFS.channel = memorychannel

 

 

 

### ruuning then 
[cloudera@localhost flume-ng]$ bin/flume-ng agent --conf ./conf/ -f conf/flume1-conf.properties -Dflume.root.logger=DEBUG,console -n twitteragent

 

 

### getting error 

2016-08-26 02:55:02,672 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:523)] Hadoop Security enabled: false
2016-08-26 02:55:02,678 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel memorychannel connected to [HDFS]
2016-08-26 02:55:02,699 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@22a98932 counterGroup:{ name:null counters:{} } }} channels:{memorychannel=org.apache.flume.channel.MemoryChannel{name: memorychannel}} }
2016-08-26 02:55:02,726 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel memorychannel
2016-08-26 02:55:02,729 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:160)] Waiting for channel: memorychannel to start. Sleeping for 500 ms
2016-08-26 02:55:02,848 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: memorychannel: Successfully registered new MBean.
2016-08-26 02:55:02,848 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: memorychannel started
2016-08-26 02:55:03,229 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink HDFS
2016-08-26 02:55:03,237 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SINK, name: HDFS: Successfully registered new MBean.
2016-08-26 02:55:03,237 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SINK, name: HDFS started
2016-08-26 02:55:03,239 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2016-08-26 02:55:33,232 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:56:03,233 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:56:33,234 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:57:03,235 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:57:33,236 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:58:03,237 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:58:33,237 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:59:03,239 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 02:59:33,240 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:00:03,241 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:00:33,242 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:01:03,243 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:01:33,244 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:02:03,245 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:02:33,246 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:03:03,247 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:03:33,248 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:04:03,249 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:04:33,251 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:05:03,253 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:05:33,254 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:06:03,255 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:06:33,256 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:07:03,257 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:07:33,258 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:08:03,259 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:08:33,260 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:09:03,261 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:09:33,262 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:10:03,264 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:10:33,265 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:11:03,266 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:11:33,267 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:12:03,269 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:12:33,270 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:13:03,271 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:13:33,272 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:14:03,273 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:14:33,275 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:15:03,276 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:15:33,277 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes
2016-08-26 03:16:03,279 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes

 

 

please help me on this asap.

Thanks

Cloudera Employee
Posts: 273
Registered: ‎01-09-2014

Re: Twitter flume ingestion

It doesn't appear there are any errors in this log. What is the error you are observing?

-pd
New Contributor
Posts: 3
Registered: ‎08-26-2016

Re: Twitter flume ingestion

Actually this is keep repeating and nothing is getting inserted , the process goes on for more than 20 -25 min but nothing happens .

 

I am not sure if this is error or not

 

Component type: SINK, name: HDFS started
2016-08-26 02:55:03,239 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2016-08-26 02:55:33,232 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume1-conf.properties for changes

 

how should i proceed furthure ?

Announcements
New solutions