question Re: Flume Authentication credentials are missing in Support Questions

Flume Authentication credentials are missing

PentaReddy — Fri, 16 Sep 2022 10:52:47 GMT

I am trying to fetch twitter data into Hdfs and i have set all the required credentials and attached is the error log, which i am getting while running flume-agent and attached is the is my conf file.

Appreciate if someone could be help me.

ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows.
java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/en/configuration.html for details
at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:215)
at twitter4j.TwitterStreamImpl.filter(TwitterStreamImpl.java:317)
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:150)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Flume.conf

Re: Flume Authentication credentials are missing

mbigelow — Tue, 03 Jan 2017 06:31:07 GMT

It thinks the credentials are missing. How did you set the OAuth consumer and token settings? Please mask the actual values but share the files and/or method they are provided to Flume.

Re: Flume Authentication credentials are missing

PentaReddy — Tue, 03 Jan 2017 06:41:07 GMT

I have logged into the twitter app and registered my app and through that i got the Oauth keys and i have placed them in the config file.

below is the conf file and i have removed my keys and replaced it as 'XXX' (masking it).

TwitterAgent.sources = Twitter
TwitterAgent.sinks = HDFS
TwitterAgent.channels = MemChannel
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerkey = XXX
TwitterAgent.sources.Twitter.consumerSecret = XXX
TwitterAgent.sources.Twitter.accessToken = XXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXX
TwitterAgent.sources.Twitter.keywords = seattle
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/user/cloudera/tweets/
TwitterAgent.sinks.HDFS.hdfs.filetype = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchsize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionalCapacity = 100

Re: Flume Authentication credentials are missing

mbigelow — Tue, 03 Jan 2017 06:58:43 GMT

TwitterAgent.sources.Twitter.consumerkey = XXX

should be

TwitterAgent.sources.Twitter.consumerKey = XXX

Re: Flume Authentication credentials are missing

PentaReddy — Tue, 03 Jan 2017 07:01:57 GMT

Is there any spelling mistake in it?

Re: Flume Authentication credentials are missing

mbigelow — Tue, 03 Jan 2017 07:03:28 GMT

The 'K' needs to be capitalized.

Re: Flume Authentication credentials are missing

PentaReddy — Tue, 03 Jan 2017 07:05:57 GMT

Ok let me fix and test it

Re: Flume Authentication credentials are missing

SandyCT — Tue, 03 Oct 2017 01:58:11 GMT

Hi Penta,

Did it work?

Actually Im facing the same issue and this is what I have used:

a1.sources.Twitter.consumerKey=XXX

a1.sources.Twitter.consumerSecret=XXX

a1.sources.Twitter.accessToken=XXX

a1.sources.Twitter.accessTokenSecret=XXX

I am trying to run the flume agent in cloudera VM.

Please advice if you or anyone know the solution.

Appreciate your suggestions/help!