I am trying to fetch twitter feed. But it is not running. It is stuck at the following response: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2022-01-30 23:46:26,038 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 2022-01-30 23:46:26,053 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/aviparna/apache-flume-1.9.0-bin/conf/flume.conf 2022-01-30 23:46:26,059 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,072 INFO conf.FlumeConfiguration: Processing:MemChannel 2022-01-30 23:46:26,072 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,072 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:MemChannel 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:MemChannel 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,073 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,074 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,075 INFO conf.FlumeConfiguration: Processing:HDFS 2022-01-30 23:46:26,075 INFO conf.FlumeConfiguration: Processing:Twitter 2022-01-30 23:46:26,075 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' has no configfilters. 2022-01-30 23:46:26,129 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent] 2022-01-30 23:46:26,132 INFO node.AbstractConfigurationProvider: Creating channels 2022-01-30 23:46:26,156 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory 2022-01-30 23:46:26,161 INFO node.AbstractConfigurationProvider: Created channel MemChannel 2022-01-30 23:46:26,161 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type com.cloudera.flume.source.TwitterSource 2022-01-30 23:46:26,274 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 2022-01-30 23:46:26,300 ERROR node.AbstractConfigurationProvider: Sink HDFS has been removed due to an error during configuration java.lang.InstantiationException: Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity. Sink: HDFS, batch size = 200000, channel MemChannel, transaction capacity = 1000 at org.apache.flume.node.AbstractConfigurationProvider.checkSinkChannelCompatibility(AbstractConfigurationProvider.java:403) at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:462) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-01-30 23:46:26,318 INFO node.AbstractConfigurationProvider: Channel MemChannel connected to [Twitter] 2022-01-30 23:46:26,323 INFO node.Application: Starting new configuration:{ sourceRunners:{Twitter=EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} }} sinkRunners:{} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name: MemChannel}} } 2022-01-30 23:46:26,339 INFO node.Application: Starting Channel MemChannel 2022-01-30 23:46:26,547 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: MemChannel: Successfully registered new MBean. 2022-01-30 23:46:26,547 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: MemChannel started 2022-01-30 23:46:26,547 INFO node.Application: Starting Source Twitter 2022-01-30 23:46:26,550 INFO twitter4j.TwitterStreamImpl: Establishing connection. 2022-01-30 23:46:27,910 INFO twitter4j.TwitterStreamImpl: Connection established. 2022-01-30 23:46:27,911 INFO twitter4j.TwitterStreamImpl: Receiving status stream.   Attaching my flume.conf file for reference: TwitterAgent.sources = Twitter TwitterAgent.channels = MemChannel TwitterAgent.sinks = HDFS # Describing/Configuring the source TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource TwitterAgent.sources.Twitter.channels = MemChannel TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxxxxxxxxxxxxx TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx TwitterAgent.sources.Twitter.keywords = bitcoin TwitterAgent.sources.Twitter.maxBatchSize = 50000 TwitterAgent.sources.Twitter.maxBatchDurationMillis = 100000 # Describing/Configuring the sink TwitterAgent.sinks.HDFS.channel=MemChannel TwitterAgent.sinks.HDFS.type=hdfs TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:9000/user/*****/tweets TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream TwitterAgent.sinks.HDFS.hdfs.writeformat=Text TwitterAgent.sinks.HDFS.hdfs.batchSize=200000 TwitterAgent.sinks.HDFS.hdfs.rollSize=0 TwitterAgent.sinks.HDFS.hdfs.rollCount = 2000000 TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600 # Describing/Configuring the channel TwitterAgent.channels.MemChannel.type = memory TwitterAgent.channels.MemChannel.capacity = 10000 TwitterAgent.channels.MemChannel.transactionCapacity = 1000 #bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1 -Dflume.root.logger=DEBUG,console ... View more