Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[Error] Stuck when run flume-ng to streaming Twitter Data into Hdfs

[Error] Stuck when run flume-ng to streaming Twitter Data into Hdfs

New Contributor

already set the other config, I'm follow the tutorial from this https://www.youtube.com/watch?v=fUesPFJ6FfE&t=444s
but when I run flume-ng, It's stuck. didn't go well like in the video. I'm using flume 1.9.0

Info: Including Hadoop libraries found via (/root/hadoop-2.9.1/bin/hadoop) for HDFS access

Info: Including Hive libraries found via () for Hive access

+ exec /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/bin/java -Xmx20m -Dflume.root.logger=DEBUG.console -cp 'conf:/usr/lib/flume/lib/*:/root/hadoop-2.9.1/etc/hadoop:/root/hadoop-2.9.1/share/hadoop/common-2.9.1/share/hadoop/common/*:/root/hadoop-2.9.1/share/hadoop/hdfs:/root/hadoop-2.9.1/share/hadoop/hdfs/lib/*:/root/hadoop-2.9.1/share/hadoop/hdfs/*:/root/hadoop-2.9.1/share/hadoop/yarn:/root/hadoop-2.9.1/share/hadoop/yoop-2.9.1/share/hadoop/yarn/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/lib/*:/root/hadoop-2.9.1/share/hadoop/mapreduce/*:/root/hadoop-2.9.1/contrib/capacity-scheduler/*.jar:/lib/*' -Djava.library.path=:/root/hadoop-2.apache.flume.node.Application --conf-file /usr/lib/flume/conf/flume-conf.properties --name TwitterAgent

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/root/hadoop-2.9.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting

19/03/29 14:16:41 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume/conf/flume-conf.properties

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 WARN conf.FlumeConfiguration: Configuration property ignored: memory = channel called ch1 on agent1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:ch1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: log-sink1 Agent: agent1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:MemChannel

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:avro-source1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:Twitter

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:HDFS

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Processing:log-sink1

19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' has no configfilters.

19/03/29 14:16:41 WARN conf.FlumeConfiguration: Agent configuration for 'agent1' has no configfilters.

19/03/29 14:16:41 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent, agent1]

19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Creating channels

19/03/29 14:16:41 INFO channel.DefaultChannelFactory: Creating instance of channel MemChannel type memory

19/03/29 14:16:41 INFO node.AbstractConfigurationProvider: Created channel MemChannel

19/03/29 14:16:41 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type org.apache.flume.source.twitter.TwitterSource

19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Source Twitter has been removed due to an error during configuration

java.lang.NullPointerException: consumer key is null

at twitter4j.TwitterBaseImpl.setOAuthConsumer(TwitterBaseImpl.java:250)

at org.apache.flume.source.twitter.TwitterSource.configure(TwitterSource.java:111)

at org.apache.flume.conf.Configurables.configure(Configurables.java:41)

at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:325)

at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105)

at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

19/03/29 14:16:41 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs

19/03/29 14:16:41 ERROR node.AbstractConfigurationProvider: Sink HDFS has been removed due to an error during configuration

java.lang.InstantiationException: Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity. Sink: HDFS, batch size = 1000, channel MemChannel, transaction capa

at org.apache.flume.node.AbstractConfigurationProvider.checkSinkChannelCompatibility(AbstractConfigurationProvider.java:403)

at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:462)

at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:106)

at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

19/03/29 14:16:41 WARN node.AbstractConfigurationProvider: Channel MemChannel has no components connected and has been removed.

19/03/29 14:16:41 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }


stuck there, after channels the font with Bold...
anyone can help? I'm newbie in Big data and still learning how to ingest data before go deep in Hadoop ecosystem.

Don't have an account?
Coming from Hortonworks? Activate your account here