Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[Error] Stuck when run flume-ng to streaming Twitter Data into Hdfs

Highlighted

[Error] Stuck when run flume-ng to streaming Twitter Data into Hdfs

New Contributor

already set the other config, I'm follow the tutorial from this https://www.youtube.com/watch?v=fUesPFJ6FfE&t=444s
but when I run flume-ng, It's stuck. didn't go well like in the video. I'm using flume 1.9.0

Info: Including Hadoop libraries found via (/root/hadoop-2.9.1/bin/hadoop) for HDFS access

Info: Including Hive libraries found via () for Hive access

+ exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/bin/java -Xmx20m -Dflume.root.logger=DEBUG.console -cp 'conf:/usr/lib/flume/lib/*:/root/hadoop-2.9.1/etc/hadoop:/root/hadoop-2.9.1/share/hadoop/common-2.9.1/share/hadoop/common/*:/root/hadoop-2.9.1/share/hadoop/hdfs:/root/hadoop-2.9.1/share/hadoop/hdfs/lib/*:/root/hadoop-2.9.1/share/hadoop/hdfs/*:/root/hadoop-2.9.1/share/hadoop/yarn:/root/hadoop-2.9.1/share/hadoop/yoop-2.9.1/share/hadoop/yarn/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/lib/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/*:/root/hadoop-2.9.1/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/root/hadoop-2.apache.flume.node.Application --conf-file /usr/lib/flume/conf/flume-conf.properties --name TwitterAgent

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/root/hadoop-2.9.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting

19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume/conf/flume-conf.properties

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 WARN conf.FlumeConfiguration: Configuration property ignored: memory = channel called ch1 on agent1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:ch1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: log-sink1 Agent: agent1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1

19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' has no configfilters.

19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'agent1' has no configfilters.

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent, agent1]

19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Creating channels

19/03/29 14:16:41 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory

19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Created channel MemChannel

19/03/29 14:16:41 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type org.apache.flume.source.twitter.TwitterSource

19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Source Twitter has been removed due to an error during configuration

java.lang.NullPointerException: consumer key is null

at twitter4j.TwitterBaseImpl.setOAuthConsumer(TwitterBaseImpl.java:250)

at org.apache.flume.source.twitter.TwitterSource.configure(TwitterSource.java:111)

at org.apache.flume.conf.Configurables.configure(Configurables.java:41)

at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:325)

at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105)

at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

19/03/29 14:16:41 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs

19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Sink HDFS has been removed due to an error during configuration

java.lang.InstantiationException: Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity. Sink: HDFS, batch size = 1000, channel MemChannel, transaction capa

at org.apache.flume.node.AbstractConfigurationProvider.checkSinkChannelCompatibility(AbstractConfigurationProvider.java:403)

at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:462)

at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106)

at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

19/03/29 14:16:41 WARN node.AbstractConfigurationProvider: Channel MemChannel has no components connected and has been removed.

19/03/29 14:16:41 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }


stuck there, after channels the font with Bold...
anyone can help? I'm newbie in Big data and still learning how to ingest data before go deep in Hadoop ecosystem.