Reply
Explorer
Posts: 15
Registered: ‎11-04-2015

Flume agent checking conf file in a loop

[ Edited ]

Hi, I am trying to get twitter feed into HDFS for certain key words. Below is my conf file

 

# Twitter conf
twittteragent.sources = Twitter
twitteragent.channels = MemChannel
twitteragent.sinks = HDFS

twitteragent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
twitteragent.sources.Twitter.channels = MemChannel
twitteragent.sources.Twitter.consumerKey = xxxx
twitteragent.sources.Twitter.consumerSecret = xxxx
twitteragent.sources.Twitter.accessToken = xxxx
twitteragent.sources.Twitter.accessTokenSecret = xxxx

twitteragent.sources.Twitter.keywords = ATTCares, hadoop, cloudera, developer

twitteragent.sinks.HDFS.channel = MemChannel
twitteragent.sinks.HDFS.type = hdfs
twitteragent.sinks.HDFS.hdfs.path = /user/cloudera/ram/tweets/
twitteragent.sinks.HDFS.hdfs.filetype = DataStream
twitteragent.sinks.HDFS.hdfs.writeFormat = Text
twitteragent.sinks.HDFS.hdfs.batchSize = 1000
twitteragent.sinks.HDFS.hdfs.rollSize = 0
twitteragent.sinks.HDFS.hdfs.rollCount = 10000

twitteragent.channels.MemChannel.type = memory
twitteragent.channels.MemChannel.capacity = 10000
twitteragent.channels.MemChannel.transactionCapacity = 100

 

When I execute this file I am getting following message in a loop without fetching any tweets. Can you please help with stopping the loop and what is the error on why it is not fetching any tweets?

 

2016-01-18 18:14:36,653 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume-twitter.conf for changes
2016-01-18 18:15:06,654 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume-twitter.conf for changes

 

Thanks in advance

Posts: 1,892
Kudos: 431
Solutions: 302
Registered: ‎07-31-2013

Re: Flume agent checking conf file in a loop

[ Edited ]

This is the internal config-update checking loop that runs every second. Its a DEBUG level log and is tied to flume internals (vs. your configuration) and is not the source of the issue.

Can you share your entire INFO+ log, if you are not seeing any data come through?


New Contributor
Posts: 2
Registered: ‎03-31-2016

Re: Flume agent checking conf file in a loop

Facing same issue ,

My flume-twitter.conf file is,

Any solution let me know

TwitterAgent
.sources = Twitter TwitterAgent.channels = MemChannel TwitterAgent.sinks = HDFS TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource TwitterAgent.sources.Twitter.channels = MemChannel TwitterAgent.sources.Twitter.consumerKey =<Key> TwitterAgent.sources.Twitter.consumerSecret =<Key> TwitterAgent.sources.Twitter.accessToken =<Key> TwitterAgent.sources.Twitter.accessTokenSecret =<Key> TwitterAgent.sources.Twitter.keywords = pig,hive,mongo,neo4j TwitterAgent.sinks.HDFS.channel = MemChannel TwitterAgent.sinks.HDFS.type = hdfsTwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/flume/tweets/ TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000 TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
Logs as below ,
vm4learning@vm4learning:~/Installations/apache-flume-1.4.0-bin$ bin/flume-ng agent --conf ./conf/ -f conf/flume-twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent
Info: Sourcing environment configuration script /home/vm4learning/Installations/apache-flume-1.4.0-bin/conf/flume-env.sh
Info: Including Hadoop libraries found via (/home/vm4learning/Installations/hadoop-1.2.1/bin/hadoop) for HDFS access
Warning: $HADOOP_HOME is deprecated.

Warning: $HADOOP_HOME is deprecated.

Info: Excluding /home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
Info: Including HBASE libraries found via (/home/vm4learning/Installations/hbase-0.94.14/bin/hbase) for HBASE access
no main manifest attribute, in /home/vm4learning/Installations/hbase-0.94.14/lib/coprocessor.jar
Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty
no main manifest attribute, in /home/vm4learning/Installations/hbase-0.94.14/lib/coprocessor.jar
Info: Excluding /home/vm4learning/Installations/hbase-0.94.14/lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /home/vm4learning/Installations/hbase-0.94.14/lib/slf4j-log4j12-1.4.3.jar from classpath
Info: Excluding /home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
+ exec /usr/lib/jvm/java-7-oracle/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/home/vm4learning/Installations/apache-flume-1.4.0-bin/conf:/home/vm4learning/Installations/apache-flume-1.4.0-bin/lib/*:/home/vm4learning/Installations/apache-flume-1.4.0-bin/flume-sources-1.0-SNAPSHOT.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/..:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../hadoop-core-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/asm-3.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/aspectjrt-1.6.11.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/aspectjtools-1.6.11.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-beanutils-1.7.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-cli-1.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-codec-1.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-collections-3.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-configuration-1.6.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-daemon-1.0.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-digester-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-el-1.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-httpclient-3.0.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-io-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-lang-2.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-logging-1.1.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-logging-api-1.0.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-math-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-net-3.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/core-3.1.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hsqldb-1.8.0.10.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jasper-compiler-5.5.12.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jasper-runtime-5.5.12.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jdeb-0.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-core-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-json-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-server-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jets3t-0.6.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jetty-6.1.26.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jetty-util-6.1.26.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsch-0.1.42.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/junit-4.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/kfs-0.2.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/log4j-1.2.15.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/mockito-all-1.8.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/oro-2.0.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/servlet-api-2.5-20081211.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/xmlenc-0.52.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:/home/vm4learning/Installations/hbase-0.94.14:/home/vm4learning/Installations/hbase-0.94.14/hbase-0.94.14.jar:/home/vm4learning/Installations/hbase-0.94.14/hbase-0.94.14-tests.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/activation-1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/asm-3.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/avro-1.5.3.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/avro-ipc-1.5.3.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-beanutils-1.7.0.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-beanutils-core-1.8.0.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-cli-1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-codec-1.4.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-collections-3.2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-configuration-1.6.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-digester-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-el-1.0.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-httpclient-3.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-io-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-lang-2.5.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-logging-1.1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-math-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-net-1.4.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/coprocessor.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/core-3.1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/guava-11.0.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/hadoop-core-1.0.4.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/high-scale-lib-1.1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/httpclient-4.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/httpcore-4.1.3.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-core-asl-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-jaxrs-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-mapper-asl-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-xc-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jamon-runtime-2.3.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jasper-compiler-5.5.23.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jasper-runtime-5.5.23.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jaxb-api-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jaxb-impl-2.2.3-1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jersey-core-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jersey-json-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jersey-server-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jettison-1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jetty-6.1.26.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jetty-util-6.1.26.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jruby-complete-1.6.5.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jsp-2.1-6.1.14.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jsp-api-2.1-6.1.14.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jsr305-1.3.9.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/junit-4.10-HBASE-1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/libthrift-0.8.0.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/log4j-1.2.16.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/metrics-core-2.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/netty-3.2.4.Final.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/phoenix-2.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/protobuf-java-2.4.0a.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/servlet-api-2.5-6.1.14.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/snappy-java-1.0.3.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/stax-api-1.0.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/velocity-1.7.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/xmlenc-0.52.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/zookeeper-3.4.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/..:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../hadoop-core-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/asm-3.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/aspectjrt-1.6.11.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/aspectjtools-1.6.11.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-beanutils-1.7.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-cli-1.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-codec-1.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-collections-3.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-configuration-1.6.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-daemon-1.0.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-digester-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-el-1.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-httpclient-3.0.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-io-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-lang-2.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-logging-1.1.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-logging-api-1.0.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-math-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-net-3.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/core-3.1.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hsqldb-1.8.0.10.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jasper-compiler-5.5.12.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jasper-runtime-5.5.12.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jdeb-0.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-core-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-json-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-server-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jets3t-0.6.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jetty-6.1.26.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jetty-util-6.1.26.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsch-0.1.42.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/junit-4.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/kfs-0.2.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/log4j-1.2.15.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/mockito-all-1.8.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/oro-2.0.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/servlet-api-2.5-20081211.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/xmlenc-0.52.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/home/vm4learning/Installations/hbase-0.94.12/lib/phoenix-2.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/conf' -Djava.library.path=:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/native/Linux-amd64-64 org.apache.flume.node.Application -f conf/flume-twitter.conf -n TwitterAgent
2016-03-31 21:52:37,719 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2016-03-31 21:52:37,733 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:78)] Configuration provider started
2016-03-31 21:52:37,738 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume-twitter.conf for changes
2016-03-31 21:52:37,779 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/flume-twitter.conf
2016-03-31 21:52:37,805 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,814 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1020)] Created context for HDFS: hdfs.rollCount
2016-03-31 21:52:37,817 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,827 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,828 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: HDFS Agent: TwitterAgent
2016-03-31 21:52:37,828 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,829 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,829 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,829 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,830 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,853 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:313)] Starting validation of configuration for agent: TwitterAgent, initial-configuration: AgentConfiguration[TwitterAgent]
SOURCES: {Twitter={ parameters:{consumerKey=kpcad8p7wNhyRedikDUwutxsC, keywords=hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing, accessToken=711960077092069377-tU5B8acE0hw1Z3TvaHMztwvjEJkAHvi, accessTokenSecret=E8k7dPFTaatNK2b7otgsZB6Hprrt8eLpNwCb2GhJhqsBV, channels=MemChannel, type=com.cloudera.flume.source.TwitterSource, consumerSecret=T3Nfp1J6ugo3PRaBOxn1g0Qqf0TVNxmST5lQ7UKrzz49ULEtXz} }}
CHANNELS: {MemChannel={ parameters:{transactionCapacity=100, capacity=10000, type=memory} }}
SINKS: {HDFS={ parameters:{hdfs.fileType=DataStream, hdfs.path=hdfs://localhost:9000/user/flume/tweets/, hdfs.batchSize=1000, hdfs.rollSize=0, hdfs.writeFormat=Text, type=hdfs, channel=MemChannel, hdfs.rollCount=10000} }}

2016-03-31 21:52:37,872 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:468)] Created channel MemChannel
2016-03-31 21:52:37,898 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:674)] Creating sink: HDFS using HDFS
2016-03-31 21:52:37,905 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:371)] Post validation configuration for TwitterAgent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[TwitterAgent]
SOURCES: {Twitter={ parameters:{consumerKey=kpcad8p7wNhyRedikDUwutxsC, keywords=hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing, accessToken=711960077092069377-tU5B8acE0hw1Z3TvaHMztwvjEJkAHvi, accessTokenSecret=E8k7dPFTaatNK2b7otgsZB6Hprrt8eLpNwCb2GhJhqsBV, channels=MemChannel, type=com.cloudera.flume.source.TwitterSource, consumerSecret=T3Nfp1J6ugo3PRaBOxn1g0Qqf0TVNxmST5lQ7UKrzz49ULEtXz} }}
CHANNELS: {MemChannel={ parameters:{transactionCapacity=100, capacity=10000, type=memory} }}
SINKS: {HDFS={ parameters:{hdfs.fileType=DataStream, hdfs.path=hdfs://localhost:9000/user/flume/tweets/, hdfs.batchSize=1000, hdfs.rollSize=0, hdfs.writeFormat=Text, type=hdfs, channel=MemChannel, hdfs.rollCount=10000} }}

2016-03-31 21:52:37,906 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:135)] Channels:MemChannel

2016-03-31 21:52:37,906 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:136)] Sinks HDFS

2016-03-31 21:52:37,906 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:137)] Sources Twitter

2016-03-31 21:52:37,907 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [TwitterAgent]
2016-03-31 21:52:37,907 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2016-03-31 21:52:37,957 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel MemChannel type memory
2016-03-31 21:52:38,009 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel MemChannel
2016-03-31 21:52:38,013 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source Twitter, type com.cloudera.flume.source.TwitterSource
2016-03-31 21:52:38,020 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:59)] Source type com.cloudera.flume.source.TwitterSource is a custom type
2016-03-31 21:52:38,270 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: HDFS, type: hdfs



TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/flume/tweets/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-collections-3.2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-configuration-1.6.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-digester-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-el-1.0.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-httpclient-3.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-io-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-lang-2.5.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-logging-1.1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-math-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/commons-net-1.4.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/coprocessor.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/core-3.1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/guava-11.0.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/hadoop-core-1.0.4.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/high-scale-lib-1.1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/httpclient-4.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/httpcore-4.1.3.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-core-asl-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-jaxrs-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-mapper-asl-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jackson-xc-1.8.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jamon-runtime-2.3.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jasper-compiler-5.5.23.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jasper-runtime-5.5.23.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jaxb-api-2.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jaxb-impl-2.2.3-1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jersey-core-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jersey-json-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jersey-server-1.8.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jettison-1.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jetty-6.1.26.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jetty-util-6.1.26.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jruby-complete-1.6.5.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jsp-2.1-6.1.14.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jsp-api-2.1-6.1.14.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/jsr305-1.3.9.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/junit-4.10-HBASE-1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/libthrift-0.8.0.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/log4j-1.2.16.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/metrics-core-2.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/netty-3.2.4.Final.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/phoenix-2.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/protobuf-java-2.4.0a.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/servlet-api-2.5-6.1.14.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/snappy-java-1.0.3.2.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/stax-api-1.0.1.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/velocity-1.7.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/xmlenc-0.52.jar:/home/vm4learning/Installations/hbase-0.94.14/lib/zookeeper-3.4.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../conf:/usr/lib/jvm/java-7-oracle/lib/tools.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/..:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../hadoop-core-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/asm-3.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/aspectjrt-1.6.11.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/aspectjtools-1.6.11.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-beanutils-1.7.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-cli-1.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-codec-1.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-collections-3.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-configuration-1.6.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-daemon-1.0.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-digester-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-el-1.0.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-httpclient-3.0.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-io-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-lang-2.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-logging-1.1.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-logging-api-1.0.4.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-math-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/commons-net-3.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/core-3.1.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/hsqldb-1.8.0.10.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jasper-compiler-5.5.12.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jasper-runtime-5.5.12.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jdeb-0.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-core-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-json-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jersey-server-1.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jets3t-0.6.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jetty-6.1.26.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jetty-util-6.1.26.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsch-0.1.42.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/junit-4.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/kfs-0.2.2.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/log4j-1.2.15.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/mockito-all-1.8.5.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/oro-2.0.8.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/servlet-api-2.5-20081211.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/xmlenc-0.52.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/home/vm4learning/Installations/hbase-0.94.12/lib/phoenix-2.1.2.jar:/home/vm4learning/Installations/hbase-0.94.14/conf' -Djava.library.path=:/home/vm4learning/Installations/hadoop-1.2.1/libexec/../lib/native/Linux-amd64-64 org.apache.flume.node.Application -f conf/flume-twitter.conf -n TwitterAgent
2016-03-31 21:52:37,719 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2016-03-31 21:52:37,733 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:78)] Configuration provider started
2016-03-31 21:52:37,738 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume-twitter.conf for changes
2016-03-31 21:52:37,779 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/flume-twitter.conf
2016-03-31 21:52:37,805 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,814 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1020)] Created context for HDFS: hdfs.rollCount
2016-03-31 21:52:37,817 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,827 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,828 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: HDFS Agent: TwitterAgent
2016-03-31 21:52:37,828 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,829 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,829 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,829 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,830 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:HDFS
2016-03-31 21:52:37,853 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:313)] Starting validation of configuration for agent: TwitterAgent, initial-configuration: AgentConfiguration[TwitterAgent]
SOURCES: {Twitter={ parameters:{consumerKey=kpcad8p7wNhyRedikDUwutxsC, keywords=hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing, accessToken=711960077092069377-tU5B8acE0hw1Z3TvaHMztwvjEJkAHvi, accessTokenSecret=E8k7dPFTaatNK2b7otgsZB6Hprrt8eLpNwCb2GhJhqsBV, channels=MemChannel, type=com.cloudera.flume.source.TwitterSource, consumerSecret=T3Nfp1J6ugo3PRaBOxn1g0Qqf0TVNxmST5lQ7UKrzz49ULEtXz} }}
CHANNELS: {MemChannel={ parameters:{transactionCapacity=100, capacity=10000, type=memory} }}
SINKS: {HDFS={ parameters:{hdfs.fileType=DataStream, hdfs.path=hdfs://localhost:9000/user/flume/tweets/, hdfs.batchSize=1000, hdfs.rollSize=0, hdfs.writeFormat=Text, type=hdfs, channel=MemChannel, hdfs.rollCount=10000} }}

2016-03-31 21:52:37,872 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:468)] Created channel MemChannel
2016-03-31 21:52:37,898 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:674)] Creating sink: HDFS using HDFS
2016-03-31 21:52:37,905 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:371)] Post validation configuration for TwitterAgent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[TwitterAgent]
SOURCES: {Twitter={ parameters:{consumerKey=kpcad8p7wNhyRedikDUwutxsC, keywords=hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing, accessToken=711960077092069377-tU5B8acE0hw1Z3TvaHMztwvjEJkAHvi, accessTokenSecret=E8k7dPFTaatNK2b7otgsZB6Hprrt8eLpNwCb2GhJhqsBV, channels=MemChannel, type=com.cloudera.flume.source.TwitterSource, consumerSecret=T3Nfp1J6ugo3PRaBOxn1g0Qqf0TVNxmST5lQ7UKrzz49ULEtXz} }}
CHANNELS: {MemChannel={ parameters:{transactionCapacity=100, capacity=10000, type=memory} }}
SINKS: {HDFS={ parameters:{hdfs.fileType=DataStream, hdfs.path=hdfs://localhost:9000/user/flume/tweets/, hdfs.batchSize=1000, hdfs.rollSize=0, hdfs.writeFormat=Text, type=hdfs, channel=MemChannel, hdfs.rollCount=10000} }}

2016-03-31 21:52:37,906 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:135)] Channels:MemChannel

2016-03-31 21:52:37,906 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:136)] Sinks HDFS

2016-03-31 21:52:37,906 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:137)] Sources Twitter

2016-03-31 21:52:37,907 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [TwitterAgent]
2016-03-31 21:52:37,907 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2016-03-31 21:52:37,957 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel MemChannel type memory
2016-03-31 21:52:38,009 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel MemChannel
2016-03-31 21:52:38,013 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source Twitter, type com.cloudera.flume.source.TwitterSource
2016-03-31 21:52:38,020 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:59)] Source type com.cloudera.flume.source.TwitterSource is a custom type
2016-03-31 21:52:38,270 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: HDFS, type: hdfs
2016-03-31 21:53:10,255 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:493)] Hadoop Security enabled: false
2016-03-31 21:53:10,271 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel MemChannel connected to [Twitter, HDFS]
2016-03-31 21:53:10,300 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{Twitter=EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} }} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5113622 counterGroup:{ name:null counters:{} } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name: MemChannel}} }
2016-03-31 21:53:10,412 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel MemChannel
2016-03-31 21:53:10,756 (lifecycleSupervisor-1-6) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: CHANNEL, name: MemChannel, registered successfully.
2016-03-31 21:53:10,758 (lifecycleSupervisor-1-6) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: MemChannel started
2016-03-31 21:53:10,759 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink HDFS
2016-03-31 21:53:10,760 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source Twitter
2016-03-31 21:53:10,768 (lifecycleSupervisor-1-8) [DEBUG - com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:124)] Setting up Twitter sample stream using consumer key kpcad8p7wNhyRedikDUwutxsC and access token 711960077092069377-tU5B8acE0hw1Z3TvaHMztwvjEJkAHvi
2016-03-31 21:53:10,813 (lifecycleSupervisor-1-7) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SINK, name: HDFS, registered successfully.
2016-03-31 21:53:10,814 (lifecycleSupervisor-1-7) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SINK, name: HDFS started
2016-03-31 21:53:10,836 (lifecycleSupervisor-1-8) [DEBUG - com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:138)] Starting up Twitter filtering...
2016-03-31 21:53:10,837 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2016-03-31 21:53:10,880 (Twitter Stream consumer-1[initializing]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] Establishing connection.
2016-03-31 21:53:10,885 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] Twitter Stream consumer-1[Establishing connection]

I have tested Keys on twitter those are good

Cloudera Employee
Posts: 275
Registered: ‎01-09-2014

Re: Flume agent checking conf file in a loop

Can you try to change the source type to the following:
org.apache.flume.source.twitter.TwitterSource

-PD
New Contributor
Posts: 2
Registered: ‎03-31-2016

Re: Flume agent checking conf file in a loop

I tried with New source you mention  getting below exception,

ollingPropertiesFileConfigurationProvider.java:142)] Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: org.apache.flume.source.twitter.TwitterSource, class: org.apache.flume.source.twitter.TwitterSource
at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.flume.source.twitter.TwitterSource
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)
... 11 more

Announcements
New solutions