How to get entire facebook feed on particular #tags using NiFi?

[#1]

Dear All,

I'm able to receive feeds from facebook using faceebook grap API on NiFi but the issue is need to get entire feeds in one shot like (next all pages ) it getting only one page .

• In facebook Grap Api itself having Next and Previous link at last few lines while clicking it goes next pages
• paging.png
• please help me to solve this

Thanks in advance

[#2]

Hi @Iyappan Gopalakrishnan

I think you could try the following approach:

GenerateFlowFile to generate one single custom flow file

UpdateAttribute to give to this flow file the URL to request

InvokeHTTP to request the data to the Facebook API based on the attributes of the flow file

Then you add two success relationships : one to continue your flow as you expect, one to extract the "next" URL using EvaluateJsonPath to update the attribute of the flow file that contains the URL to request and send back this flow file to the InvokeHTTP.

This way you'll loop over the "next" pages to get all the data.

This probably needs to be adjusted with the specifics of Facebook API but this is an idea. If this is not helping and if you can share a template of your workflow, I may have a look.

Hope this helps.

[#3]

Hi @Pierre Villard

I tried as u mentioned and able to get next page link but not able to give url of next page link into Update attribute

• i shared my flow image

• Update Attribute image

  

• Invoke Http properties

  

• EvaluateJsonPath properties

  

• my problem is how to connect this next page url into UpdateAttribute and make loop to get next all pages because i gave direct Url in UpdateAttribute property.
• i tried to change property name next as target and connect to UpdateAttribute but not able to assign because already having direct Url.

Please Help me to solve this issue

[#4]

Hi @Iyappan Gopalakrishnan

If $.paging.next contains the exact url you want to request, then in EvaluateJsonPath instead of naming your attribute 'next', you could directly call it 'target' to update the value of 'target' in the FlowFile. Then you directly link your EvaluateJsonPath processor back to InvokeHTTP processor.

You may also want to use a RouteOnAttribute processor to have an exit path depending of the value of 'target' when you reach the last page.

Let me know if it helps.

[#5]

Thanks @Pierre Villard

• i changed 'next' as 'target' and connect to UpdateAttribute but not able to assign because already having target as direct Url.
• how to assign "EvaluateJsonPath" value "target' into UpdateAttribute .

[#6]

I'd rather directly link the 'matched' relationship from EvaluateJsonPath to InvokeHTTP. You don't want to use the UpdateAttribute anymore since you have updated the 'target' value. Also you should change the property 'destination' to 'flow file attributes' in your EvaluateJsonPath processor.

[#7]

Please share your template, I'll have a look.

[#8]

I am not able to execute the workflow because the token is no longer valid:

{"error":{"message":"Error validating access token: Session has expired on Tuesday, 02-Aug-16 03:00:00 PDT. The current time is Tuesday, 02-Aug-16 04:21:58 PDT.","type":"OAuthException","code":190,"error_subcode":463,"fbtrace_id":"GKN7NpU0NhS"}}

[#9]

Thanks @Pierre Villard

• Still not able to receive next pages and can see some failure logs.

• can i share my template

[#10]

does it get resolved? if yes then how? plz share @Pierre Villard