Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to get entire facebook feed on particular #tags using NiFi?

How to get entire facebook feed on particular #tags using NiFi?

Explorer

Dear All,

I'm able to receive feeds from facebook using faceebook grap API on NiFi but the issue is need to get entire feeds in one shot like (next all pages ) it getting only one page .

  • In facebook Grap Api itself having Next and Previous link at last few lines while clicking it goes next pages
  • paging.png
  • please help me to solve this

Thanks in advance

9 REPLIES 9
Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

Hi @Iyappan Gopalakrishnan

I think you could try the following approach:

GenerateFlowFile to generate one single custom flow file

UpdateAttribute to give to this flow file the URL to request

InvokeHTTP to request the data to the Facebook API based on the attributes of the flow file

Then you add two success relationships : one to continue your flow as you expect, one to extract the "next" URL using EvaluateJsonPath to update the attribute of the flow file that contains the URL to request and send back this flow file to the InvokeHTTP.

This way you'll loop over the "next" pages to get all the data.

This probably needs to be adjusted with the specifics of Facebook API but this is an idea. If this is not helping and if you can share a template of your workflow, I may have a look.

Hope this helps.

Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

Explorer

Hi @Pierre Villard

I tried as u mentioned and able to get next page link but not able to give url of next page link into Update attribute

  • i shared my flow image

6244-p.png

  • Update Attribute image

    6245-p1.png

  • Invoke Http properties

    6246-p2.png

  • EvaluateJsonPath properties

    6247-p3.png

  • my problem is how to connect this next page url into UpdateAttribute and make loop to get next all pages because i gave direct Url in UpdateAttribute property.
  • i tried to change property name next as target and connect to UpdateAttribute but not able to assign because already having direct Url.

Please Help me to solve this issue

Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

Hi @Iyappan Gopalakrishnan

If $.paging.next contains the exact url you want to request, then in EvaluateJsonPath instead of naming your attribute 'next', you could directly call it 'target' to update the value of 'target' in the FlowFile. Then you directly link your EvaluateJsonPath processor back to InvokeHTTP processor.

You may also want to use a RouteOnAttribute processor to have an exit path depending of the value of 'target' when you reach the last page.

Let me know if it helps.

Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

Explorer

Thanks @Pierre Villard

  • i changed 'next' as 'target' and connect to UpdateAttribute but not able to assign because already having target as direct Url.
  • how to assign "EvaluateJsonPath" value "target' into UpdateAttribute .
Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

I'd rather directly link the 'matched' relationship from EvaluateJsonPath to InvokeHTTP. You don't want to use the UpdateAttribute anymore since you have updated the 'target' value. Also you should change the property 'destination' to 'flow file attributes' in your EvaluateJsonPath processor.

Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

Please share your template, I'll have a look.

Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

I am not able to execute the workflow because the token is no longer valid:

{"error":{"message":"Error validating access token: Session has expired on Tuesday, 02-Aug-16 03:00:00 PDT. The current time is Tuesday, 02-Aug-16 04:21:58 PDT.","type":"OAuthException","code":190,"error_subcode":463,"fbtrace_id":"GKN7NpU0NhS"}}

Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

Explorer

Thanks @Pierre Villard

  • Still not able to receive next pages and can see some failure logs.
  • can i share my template
Highlighted

Re: How to get entire facebook feed on particular #tags using NiFi?

does it get resolved? if yes then how? plz share @Pierre Villard

Don't have an account?
Coming from Hortonworks? Activate your account here