already set the other config, I'm follow the tutorial from this https://www.youtube.com/watch?v=fUesPFJ6FfE&t=444s
but when I run flume-ng, It's stuck. didn't go well like in the video. I'm using flume 1.9.0
Info: Including Hadoop libraries found via (/root/hadoop-2.9.1/bin/hadoop) for HDFS access
Info: Including Hive libraries found via () for Hive access
+ exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/bin/java -Xmx20m -Dflume.root.logger=DEBUG.console -cp 'conf:/usr/lib/flume/lib/*:/root/hadoop-2.9.1/etc/hadoop:/root/hadoop-2.9.1/share/hadoop/common-2.9.1/share/hadoop/common/*:/root/hadoop-2.9.1/share/hadoop/hdfs:/root/hadoop-2.9.1/share/hadoop/hdfs/lib/*:/root/hadoop-2.9.1/share/hadoop/hdfs/*:/root/hadoop-2.9.1/share/hadoop/yarn:/root/hadoop-2.9.1/share/hadoop/yoop-2.9.1/share/hadoop/yarn/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/lib/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/*:/root/hadoop-2.9.1/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/root/hadoop-2.apache.flume.node.Application --conf-file /usr/lib/flume/conf/flume-conf.properties --name TwitterAgent
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-2.9.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume/conf/flume-conf.properties
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 WARN conf.FlumeConfiguration: Configuration property ignored: memory = channel called ch1 on agent1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:ch1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: log-sink1 Agent: agent1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1
19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' has no configfilters.
19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'agent1' has no configfilters.
19/03/29 14:16:41 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent, agent1]
19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Creating channels
19/03/29 14:16:41 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory
19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Created channel MemChannel
19/03/29 14:16:41 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type org.apache.flume.source.twitter.TwitterSource
19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Source Twitter has been removed due to an error during configuration
java.lang.NullPointerException: consumer key is null
at twitter4j.TwitterBaseImpl.setOAuthConsumer(TwitterBaseImpl.java:250)
at org.apache.flume.source.twitter.TwitterSource.configure(TwitterSource.java:111)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:325)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/03/29 14:16:41 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs
19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Sink HDFS has been removed due to an error during configuration
java.lang.InstantiationException: Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity. Sink: HDFS, batch size = 1000, channel MemChannel, transaction capa
at org.apache.flume.node.AbstractConfigurationProvider.checkSinkChannelCompatibility(AbstractConfigurationProvider.java:403)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:462)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/03/29 14:16:41 WARN node.AbstractConfigurationProvider: Channel MemChannel has no components connected and has been removed.
19/03/29 14:16:41 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
stuck there, after channels the font with Bold...
anyone can help? I'm newbie in Big data and still learning how to ingest data before go deep in Hadoop ecosystem.
