I already set the other config, I'm following the tutorial from this https://www.youtube.com/watch?v=fUesPFJ6FfE&t=444s to streaming twitter data to hdfs but when I run flume-ng, It's stuck. didn't go well like in the video. I'm using flume 1.9.0 flume1.png flume2.png flume3.png after flume3.png, there's nothing happen, just like that stuck forever. anyone know the problem, please help me. thanks ... View more

already set the other config, I'm follow the tutorial from this https://www.youtube.com/watch?v=fUesPFJ6FfE&t=444s but when I run flume-ng, It's stuck. didn't go well like in the video. I'm using flume 1.9.0 Info: Including Hadoop libraries found via (/root/hadoop-2.9.1/bin/hadoop) for HDFS access Info: Including Hive libraries found via () for Hive access + exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/bin/java -Xmx20m -Dflume.root.logger=DEBUG.console -cp 'conf:/usr/lib/flume/lib/*:/root/hadoop-2.9.1/etc/hadoop:/root/hadoop-2.9.1/share/hadoop/common-2.9.1/share/hadoop/common/*:/root/hadoop-2.9.1/share/hadoop/hdfs:/root/hadoop-2.9.1/share/hadoop/hdfs/lib/*:/root/hadoop-2.9.1/share/hadoop/hdfs/*:/root/hadoop-2.9.1/share/hadoop/yarn:/root/hadoop-2.9.1/share/hadoop/yoop-2.9.1/share/hadoop/yarn/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/lib/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/*:/root/hadoop-2.9.1/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/root/hadoop-2.apache.flume.node.Application --conf-file /usr/lib/flume/conf/flume-conf.properties --name TwitterAgent SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/root/hadoop-2.9.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume/conf/flume-conf.properties 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 WARN conf.FlumeConfiguration: Configuration property ignored: memory = channel called ch1 on agent1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:ch1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: log-sink1 Agent: agent1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1 19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' has no configfilters. 19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'agent1' has no configfilters. 19/03/29 14:16:41 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent, agent1] 19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Creating channels 19/03/29 14:16:41 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory 19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Created channel MemChannel 19/03/29 14:16:41 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type org.apache.flume.source.twitter.TwitterSource 19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Source Twitter has been removed due to an error during configuration java.lang.NullPointerException: consumer key is null at twitter4j.TwitterBaseImpl.setOAuthConsumer(TwitterBaseImpl.java:250) at org.apache.flume.source.twitter.TwitterSource.configure(TwitterSource.java:111) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:325) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/03/29 14:16:41 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Sink HDFS has been removed due to an error during configuration java.lang.InstantiationException: Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity. Sink: HDFS, batch size = 1000, channel MemChannel, transaction capa at org.apache.flume.node.AbstractConfigurationProvider.checkSinkChannelCompatibility(AbstractConfigurationProvider.java:403) at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:462) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/03/29 14:16:41 WARN node.AbstractConfigurationProvider: Channel MemChannel has no components connected and has been removed. 19/03/29 14:16:41 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} } stuck there, after channels the font with Bold... anyone can help? I'm newbie in Big data and still learning how to ingest data before go deep in Hadoop ecosystem. ... View more