Support Questions
Find answers, ask questions, and share your expertise

Flume Authentication credentials are missing

Rising Star

I am trying to fetch twitter data into Hdfs and  i have set all the required credentials and attached is the error log, which i am getting while running flume-agent and attached is the is my conf file.


Appreciate if someone could be help me.


ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.IllegalStateException: Authentication credentials are missing. See for details
at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(
at twitter4j.TwitterStreamImpl.filter(
at com.cloudera.flume.source.TwitterSource.start(
at org.apache.flume.source.EventDrivenSourceRunner.start(
at org.apache.flume.lifecycle.LifecycleSupervisor$
at java.util.concurrent.Executors$
at java.util.concurrent.FutureTask.runAndReset(
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
at java.util.concurrent.ScheduledThreadPoolExecutor$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$


Accepted Solutions

Rising Star
Ok let me fix and test it

View solution in original post



It thinks the credentials are missing.  How did you set the OAuth consumer and token settings?  Please mask the actual values but share the files and/or method they are provided to Flume.

Rising Star

I have logged into the twitter app and registered my app and through that i got the Oauth keys and i have placed them in the config file.


below is the conf file and i have removed my keys and replaced it as 'XXX' (masking it).


TwitterAgent.sources = Twitter
TwitterAgent.sinks = HDFS
TwitterAgent.channels = MemChannel
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerkey = XXX
TwitterAgent.sources.Twitter.consumerSecret = XXX
TwitterAgent.sources.Twitter.accessToken = XXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXX
TwitterAgent.sources.Twitter.keywords = seattle = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/user/cloudera/tweets/
TwitterAgent.sinks.HDFS.hdfs.filetype = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchsize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionalCapacity = 100

TwitterAgent.sources.Twitter.consumerkey = XXX

should be

TwitterAgent.sources.Twitter.consumerKey = XXX

Rising Star
Is there any spelling mistake in it?

The 'K' needs to be capitalized.

Rising Star
Ok let me fix and test it

View solution in original post


Hi Penta,

Did it work?


Actually Im facing the same issue and this is what I have used:






I am trying to run the flume agent in cloudera VM.


Please advice if you or anyone know the solution.

Appreciate your suggestions/help!