How to set Twitter OAuth attributes to InvokeHttp processor for fetching twitter user profiles

[#1]

I use the InvokeHttp processor to fetch twitter user profiles from the url https://api.twitter.com/1.1/users/lookup.json?screen_name=twitterapi,twitter and my Nifi flow is GetFile(which sends screen_name, consumer_key, acess_token) -> updateAttribute(sets the mime.type) ->InvokeHttp(sets the url) -> PutFile. But i get the bad authentication data {"errors":[{"code":215,"message":"Bad Authentication data."}]}. How to set the oauth atrributes to the twitter url? Please help

[#2 · Accepted solution]

I would recommend you the following links:

- https://dev.twitter.com/rest/reference/get/users/lookup

- https://dev.twitter.com/oauth/overview

With the first link, if you are connected on your Twitter account, you will see a part "OAuth Signature Generator" that will create for you all the HTTP header properties you must add in your request (as an example). You may also want to have a look to official documentation about OAuth (http://oauth.net/core/1.0a/) to understand how the properties are constructed.

Let me know if you still have issues.

[#3]

I had a closer look on this, and I think your best option is to use an ExecuteScript processor to first create the parameter string, then the signature base string and, at the end, the Authorization string you need to set in the HTTP header properties of your InvokeHTTP processor.

Your flow would be something like:

GenerateFlowFile (to create an empty flow file) -> UpdateAttribute (to define all the parameters you need for OAuth) -> ExecuteScript (to compute the Authorization string) -> InvokeHTTP (with a custom property to set Authorization header property) -> PutFile (to get the result).

Regarding OAuth parameters, you will need to use Expression language for some parameters:

- timestamp with now() and subtring() to get seconds and not milliseconds.

- nonce can be generated using the UUID method

I wrote an article here to detail a solution where I use ExecuteScript processor and Groovy to perform the request you are looking for.

[#4]

@Pierre Villard, Thank you so much for your help. The post is really useful. I imported your template, But How to execute it ? when i run the template with the keys and tokens defined in update attribute the execute script processor throws me an exception. Please find attached exception.screen-shot-2016-04-12-at-14210-pm.png Please let me know.

[#5]

My bad! I tried to clean the code and introduced an error... I updated the link to the template in the article and the gist as well (you can copy/paste the code from gist to the processor if you don't want to start over from the template). It should now be OK!

[#6]

@Pierre Villard, i have a requirement, that i need to dynamically populate the screen_name in the url https://api.twitter.com/1.1/users/lookup.json?screen_name=twitterapi,twitter with the values from the hive table. Can you suggest me how can i implement with Nifi? Thanks.

[#7]

One option would be to get the content of your Hive table using a processor as the content of your FlowFile, then you could use an ExecuteScript processor to transform the content of your FlowFile as a string listing your usernames (screen_name=user1,user2,etc) and update the 'target' attribute as currently used at the begining of your current data flow.

[#8]

@Pierre Villard

Hi, as you suggested above, i got the content of the Hive table using FetchHDFS processor and sending the flowfile to a custom processor which needs to transform the flowfile content into a String listing of usernames. I am stuck here, how to loop thru the content of flowfile to fetch column1, column 3 eg SARAH ABDELSAYED) into a string listing for screen_name. I am using flowFile.getAttributes(), but could not see the log statements for the map. How to loop the content of flowfile to fetch specific columns. Thank you. My example flowfile content is attached.screenshot-7.png

[#9]

I'd suggest you to have a look at this post: http://funnifi.blogspot.co.uk/2016/02/executescript-processor-replacing-flow.html

It'll show you how to play around with the flow file content using ExecuteScript processor. Once you have your full string, you can put it as an attribute of the FlowFile or keep it as new content (it depends of what you have in mind).

Hope that helps.

[#10]

Thank you. I will look into that.