Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

GetHTTP making multiple api calls with changing parameter

avatar
Explorer

Hi i'm needing to make an api call for data pertaining to each day in a calendar year. I don't want to do this manually as it will be a laborious task and i will no doubt make a mistake at some stage. The api request is as follows:

http://api.wunderground.com/api/dfe2f11fb7108c21/history_20170101/q/CA/EWR.json

with the date parameter needing to change for each request. Is there a way to automate this process.

Thanks in advance.

1 ACCEPTED SOLUTION

avatar
Master Guru

@Mark McGowan

Use GenerateFlowFile processor with Invoke Http processor to get date dynamically.

GenerateflowFile configs:-

64936-generateff.png

in this processor add new property as

dd

${now():format("yyyyMMdd")} //Gives today's date and added as flowfile attribute with dd as name

InvokeHttp Processor:-

64937-invokehttp.png

Change the remote url in Invoke http processor with

http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

Use Response relationship from invoke http processor and we are using dd attribute value in the url request, now you don't need to change the url daily.

Schedule the generate flowfile processor to run daily so that invoke http processor will run dynamically.

Flow:-

64938-flow.png

View solution in original post

6 REPLIES 6

avatar
Master Guru

@Mark McGowan

Use GenerateFlowFile processor with Invoke Http processor to get date dynamically.

GenerateflowFile configs:-

64936-generateff.png

in this processor add new property as

dd

${now():format("yyyyMMdd")} //Gives today's date and added as flowfile attribute with dd as name

InvokeHttp Processor:-

64937-invokehttp.png

Change the remote url in Invoke http processor with

http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

Use Response relationship from invoke http processor and we are using dd attribute value in the url request, now you don't need to change the url daily.

Schedule the generate flowfile processor to run daily so that invoke http processor will run dynamically.

Flow:-

64938-flow.png

avatar
Explorer
@Shu

Thanks for your prompt response.

I can see how that would work if i was making calls for the current day as each day progresses but i need to make the calls with historical dates i.e. start of at 2017/01/01 next call is 2017/01/02 all the way to 2017/12/31.

Is this possible?

avatar
Master Guru
@Mark McGowan

Yes, we can do that for this case we need to fork invokehttp processor response relation to fork 1 response goes for processing, 2 response relation will goes to RouteOnattribute processor to check is the date is in range or not. If in range then we are updating the date and again going to invoke http processor.

Loop:-

64945-loop.png

UpdateAttribute processor configs:-

dd

${dd:isNull():ifElse('20170101','${dd:toDate("yyyyMMdd"):toNumber():plus(100800000):format("yyyyMMdd")}')}

Checking dd attribute value if the value is null then 20170101 for the first run we are going to have this value.

if not null i.e from second run we are adding 100800000 milliseconds(i.e 28 hrs because my server configured with est and we need to consider day light saving we need to consider).

Invoke Http processor configs:-

Remote URL

http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

this won’t change because we are modifying dd attribute value in Update Attribute processor.

RouteOnAttribute configs:-

In this processor we need to add new property

in range

${dd:lt('20171231')}

Checking dd attribute value less than 20171231 if the value is less than then flowfile will goes to

1.Update Attribute processor and updating the dd value(+1 day)
2.then to invoke http processor fork response
2.1.first-fork goes to Routeonattribute processor
2.2 second-fork goes for processing

i have attached my flow xml, use that for reference and make changes as per your use cases..

loop-invokehttp-181963.xm

avatar
Explorer

@Shu

Thank you sir i really appreciate your help, although i am having trouble loading your template, i am getting the following error!?

templateerror.png

avatar
Master Guru
@Mark McGowan

Try this template once

181963-loop-invokehttp.xml

if you are still facing issues then i have added all processor configs below

Flow:-

64949-flow.png

Generate flowfile leave as default and schedule by using cron (or) timer driven and running on primary node.

Update Attribute Configs:-

64950-updateattribute.png

Add new property

dd

${dd:isNull():ifElse('20170101','${dd:toDate("yyyyMMdd"):toNumber():plus(100800000):format("yyyyMMdd")}')}

Invoke http configs:-

64951-ihttp.png

Remote URL
http://api.wunderground.com/api/dfe2f11fb7108c21/history_${dd}/q/CA/EWR.json

Handle connection timeout issues and another relationships in invoke http processor(no retry..etc)

RouteOnAttribute:-

64953-roa.png

Add new property as

in range

${dd:lt('20171231')}

Connect all the relations as shown in the flow screenshot.

Let us know if you are having issues..!!

avatar
Master Guru
@Mark McGowan

Does the answer helped to resolve your issue, Click on Accept button below to accept the answer,to close the thread and it would be great help to Community users to find solution quickly for these kind of issues.