Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume Authentication credentials are missing

Solved Go to solution

Flume Authentication credentials are missing

Rising Star

I am trying to fetch twitter data into Hdfs and  i have set all the required credentials and attached is the error log, which i am getting while running flume-agent and attached is the is my conf file.

 

Appreciate if someone could be help me.

 

ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/en/configuration.html for details
at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:215)
at twitter4j.TwitterStreamImpl.filter(TwitterStreamImpl.java:317)
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:150)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)flume.conf.PNGFlume.conf

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Flume Authentication credentials are missing

Rising Star
Ok let me fix and test it
7 REPLIES 7

Re: Flume Authentication credentials are missing

Champion

It thinks the credentials are missing.  How did you set the OAuth consumer and token settings?  Please mask the actual values but share the files and/or method they are provided to Flume.

Re: Flume Authentication credentials are missing

Rising Star

I have logged into the twitter app and registered my app and through that i got the Oauth keys and i have placed them in the config file.

 

below is the conf file and i have removed my keys and replaced it as 'XXX' (masking it).

 

TwitterAgent.sources = Twitter
TwitterAgent.sinks = HDFS
TwitterAgent.channels = MemChannel
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerkey = XXX
TwitterAgent.sources.Twitter.consumerSecret = XXX
TwitterAgent.sources.Twitter.accessToken = XXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXX
TwitterAgent.sources.Twitter.keywords = seattle
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/user/cloudera/tweets/
TwitterAgent.sinks.HDFS.hdfs.filetype = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchsize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionalCapacity = 100

Re: Flume Authentication credentials are missing

Champion
TwitterAgent.sources.Twitter.consumerkey = XXX

should be

TwitterAgent.sources.Twitter.consumerKey = XXX

Re: Flume Authentication credentials are missing

Rising Star
Is there any spelling mistake in it?

Re: Flume Authentication credentials are missing

Champion
The 'K' needs to be capitalized.

Re: Flume Authentication credentials are missing

Rising Star
Ok let me fix and test it

Re: Flume Authentication credentials are missing

Explorer

Hi Penta,

Did it work?

 

Actually Im facing the same issue and this is what I have used:

a1.sources.Twitter.consumerKey=XXX

a1.sources.Twitter.consumerSecret=XXX

a1.sources.Twitter.accessToken=XXX

a1.sources.Twitter.accessTokenSecret=XXX

 

I am trying to run the flume agent in cloudera VM.

 

Please advice if you or anyone know the solution.

Appreciate your suggestions/help!

Don't have an account?
Coming from Hortonworks? Activate your account here