Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

FLUME THROWING EXCEPTION

FLUME THROWING EXCEPTION

Master Collaborator

the following command throws exception , i have installed latest flume (1.6.0  ),  I have also tried installing the 

flume-sources-1.0-SNAPSHOT.jar file by doing mvn install in  cdh-twitter-example-master  folder and then copying it to $FLUME_HOME/lib/   with no luck. 

 

flume-ng agent --conf-file twitter-to-hdfs.properties --name agent1 --conf $FLUME_HOME/conf/twitter.conf -Dflume.root.logger=WARN,console -Dtwitter4j.http.proxyHost=dotatofwproxy.tolls.dot.state.fl.us -Dtwitter4j.http.proxyPort=8080

 

16/09/08 11:46:58 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: sink1 started
16/09/08 11:46:58 INFO twitter4j.TwitterStreamImpl: Connection established.
16/09/08 11:46:58 INFO twitter4j.TwitterStreamImpl: Receiving status stream.
Exception in thread "Twitter4J Async Dispatcher[0]" java.lang.NoSuchMethodError: twitter4j.Status.getRetweetCount()J
        at org.apache.flume.source.twitter.TwitterSource.extractRecord(TwitterSource.java:248)
        at org.apache.flume.source.twitter.TwitterSource.onStatus(TwitterSource.java:157)
        at twitter4j.StatusStreamImpl.onStatus(StatusStreamImpl.java:75)

6 REPLIES 6

Re: FLUME THROWING EXCEPTION

Super Collaborator
Can you please provide the flume config that you are using?

Also, you may run into issues if you are trying to copy older jar files into the flume classpath. Flume already provides native support for the twitter source.

-pd

Re: FLUME THROWING EXCEPTION

Master Collaborator

are you talking about $FLUME_HOME/conf/flume-conf.properties file?   here are its contents

 

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink

# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq

# The channel can be defined as follows.
agent.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.loggerSink.type = logger

#Specify the channel the sink should use
agent.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100
[root@hadoop1 conf]#

Re: FLUME THROWING EXCEPTION

Master Collaborator

how can i debug this issue ?

anyone?

Re: FLUME THROWING EXCEPTION

Master Collaborator

can anyone please help ? moderators?

Re: FLUME THROWING EXCEPTION

Super Collaborator
Your flume agent is configured to use this:
twitter-to-hdfs.properties

Please provide the contents of that (and obfuscate the twitter passwords)

-pd

Re: FLUME THROWING EXCEPTION

Master Collaborator

first of all thanks for replying.  below is the properties file

and the command and error i am getting

 

flume-ng agent --conf-file twitter-to-hdfs.properties --name agent1 -Dflume.root.logger=WARN,console -Dtwitter4j.http.proxyHost=dotatofwproxy.tolls.dot.state.fl.us -Dtwitter4j.http.proxyPort=8080

16/09/16 11:55:43 ERROR node.PollingPropertiesFileConfigurationProvider: Unhandled error
java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
 

 

 

[root@hadoop1 ~]# more twitter-to-hdfs.properties
agent1.sources =source1
agent1.sinks = sink1
agent1.channels = channel1


agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1

agent1.sources.source1.type = org.apache.flume.source.twitter.TwitterSource
agent1.sources.source1.consumerKey = xxxxxxxxxxxxx
agent1.sources.source1.consumerSecret = xxxxxxxxxxxxxxxxxxx
agent1.sources.source1.accessToken = xxxxxxxxxxxxxxxxxxxxxxxx
agent1.sources.source1.accessTokenSecret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
agent1.sources.source1.keywords = @Hillary, @Clinton


agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = /user/flume/tweets
agent1.sinks.sink1.hdfs.filePrefix = events
agent1.sinks.sink1.hdfs.fileSuffix = .log
agent1.sinks.sink1.hdfs.inUsePrefix = _
agent1.sinks.sink1.hdfs.fileType = DataStream


agent1.channels.channel1.type = file