GetHTTP making multiple api calls with changing parameter

[#1]

Hi i'm needing to make an api call for data pertaining to each day in a calendar year. I don't want to do this manually as it will be a laborious task and i will no doubt make a mistake at some stage. The api request is as follows:

http://api.wunderground.com/api/dfe2f11fb7108c21/history_20170101/q/CA/EWR.json

with the date parameter needing to change for each request. Is there a way to automate this process.

Thanks in advance.

[#2 · Accepted solution]

@Mark McGowan

Use GenerateFlowFile processor with Invoke Http processor to get date dynamically.

GenerateflowFile configs:-

in this processor add new property as

dd

${now():format("yyyyMMdd")} //Gives today's date and added as flowfile attribute with dd as name

InvokeHttp Processor:-

Change the remote url in Invoke http processor with

http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

Use Response relationship from invoke http processor and we are using dd attribute value in the url request, now you don't need to change the url daily.

Schedule the generate flowfile processor to run daily so that invoke http processor will run dynamically.

Flow:-

[#3]

@Shu

Thanks for your prompt response.

I can see how that would work if i was making calls for the current day as each day progresses but i need to make the calls with historical dates i.e. start of at 2017/01/01 next call is 2017/01/02 all the way to 2017/12/31.

Is this possible?

[#4]

@Mark McGowan

Yes, we can do that for this case we need to fork invokehttp processor response relation to fork 1 response goes for processing, 2 response relation will goes to RouteOnattribute processor to check is the date is in range or not. If in range then we are updating the date and again going to invoke http processor.

Loop:-

UpdateAttribute processor configs:-

dd

${dd:isNull():ifElse('20170101','${dd:toDate("yyyyMMdd"):toNumber():plus(100800000):format("yyyyMMdd")}')}

Checking dd attribute value if the value is null then 20170101 for the first run we are going to have this value.

if not null i.e from second run we are adding 100800000 milliseconds(i.e 28 hrs because my server configured with est and we need to consider day light saving we need to consider).

Invoke Http processor configs:-

Remote URL

http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

this won’t change because we are modifying dd attribute value in Update Attribute processor.

RouteOnAttribute configs:-

In this processor we need to add new property

in range

${dd:lt('20171231')}

Checking dd attribute value less than 20171231 if the value is less than then flowfile will goes to

1.Update Attribute processor and updating the dd value(+1 day)
2.then to invoke http processor fork response 
2.1.first-fork goes to Routeonattribute processor
2.2 second-fork goes for processing

i have attached my flow xml, use that for reference and make changes as per your use cases..

loop-invokehttp-181963.xm

[#5]

@Shu

Thank you sir i really appreciate your help, although i am having trouble loading your template, i am getting the following error!?

templateerror.png

[#6]

@Mark McGowan

Try this template once

181963-loop-invokehttp.xml

if you are still facing issues then i have added all processor configs below

Flow:-

Generate flowfile leave as default and schedule by using cron (or) timer driven and running on primary node.

Update Attribute Configs:-

Add new property

dd

${dd:isNull():ifElse('20170101','${dd:toDate("yyyyMMdd"):toNumber():plus(100800000):format("yyyyMMdd")}')}

Invoke http configs:-

Remote URL

http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

Handle connection timeout issues and another relationships in invoke http processor(no retry..etc)

RouteOnAttribute:-

Add new property as

in range

${dd:lt('20171231')}

Connect all the relations as shown in the flow screenshot.

Let us know if you are having issues..!!

[#7]

@Mark McGowan

Does the answer helped to resolve your issue, Click on Accept button below to accept the answer,to close the thread and it would be great help to Community users to find solution quickly for these kind of issues.