Reply
Explorer
Posts: 11
Registered: ‎12-21-2016
Accepted Solution

Flume Authentication credentials are missing

I am trying to fetch twitter data into Hdfs and  i have set all the required credentials and attached is the error log, which i am getting while running flume-agent and attached is the is my conf file.

 

Appreciate if someone could be help me.

 

ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/en/configuration.html for details
at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:215)
at twitter4j.TwitterStreamImpl.filter(TwitterStreamImpl.java:317)
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:150)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Flume.confFlume.conf

Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Flume Authentication credentials are missing

It thinks the credentials are missing.  How did you set the OAuth consumer and token settings?  Please mask the actual values but share the files and/or method they are provided to Flume.

Explorer
Posts: 11
Registered: ‎12-21-2016

Re: Flume Authentication credentials are missing

I have logged into the twitter app and registered my app and through that i got the Oauth keys and i have placed them in the config file.

 

below is the conf file and i have removed my keys and replaced it as 'XXX' (masking it).

 

TwitterAgent.sources = Twitter
TwitterAgent.sinks = HDFS
TwitterAgent.channels = MemChannel
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerkey = XXX
TwitterAgent.sources.Twitter.consumerSecret = XXX
TwitterAgent.sources.Twitter.accessToken = XXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXX
TwitterAgent.sources.Twitter.keywords = seattle
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/user/cloudera/tweets/
TwitterAgent.sinks.HDFS.hdfs.filetype = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchsize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionalCapacity = 100

Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Flume Authentication credentials are missing

TwitterAgent.sources.Twitter.consumerkey = XXX

should be

TwitterAgent.sources.Twitter.consumerKey = XXX
Explorer
Posts: 11
Registered: ‎12-21-2016

Re: Flume Authentication credentials are missing

Is there any spelling mistake in it?
Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Flume Authentication credentials are missing

The 'K' needs to be capitalized.
Explorer
Posts: 11
Registered: ‎12-21-2016

Re: Flume Authentication credentials are missing

Ok let me fix and test it
Highlighted
New Contributor
Posts: 5
Registered: ‎10-02-2017

Re: Flume Authentication credentials are missing

Hi Penta,

Did it work?

 

Actually Im facing the same issue and this is what I have used:

a1.sources.Twitter.consumerKey=XXX

a1.sources.Twitter.consumerSecret=XXX

a1.sources.Twitter.accessToken=XXX

a1.sources.Twitter.accessTokenSecret=XXX

 

I am trying to run the flume agent in cloudera VM.

 

Please advice if you or anyone know the solution.

Appreciate your suggestions/help!

Announcements