Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Add last execution time with getHTTP processor to REST API URL as updated since parameter

Highlighted

Add last execution time with getHTTP processor to REST API URL as updated since parameter

New Contributor

Hey everybody,

I have a small question to add the last execution time of my getHttp processor to the URL as updated since parameter. The reason is the following:

I have a REST API which accepts a URL parameter called updated_since. With this parameter I can get a list of objects as JSON to work with them. The URL has a structure like this: https://api.someurl.com/api/v2/clients?updated_since=2017-12-11T14:23:52Z&access_token=xxxxxxxxxxxxx...

Now I want to add the last execution time to this url after updated_since=?????

Has somebody an idea how to realize this? I searched for this but didn't find a solution for this issue.

Thanks a lot for your help.
Michael

5 REPLIES 5

Re: Add last execution time with getHTTP processor to REST API URL as updated since parameter

Super Guru
@Michael Hannekum
if you are having lastexecutiontime attribute associated with the flowfile then use UpdataAttribute processor add the new property

url as

${url:replaceAll('updated_since=(.*)(?=∾cess_token)', 'updated_since=${lastexecutiontime}')} //in this expression we are using replaceAll function and we are having lastexecutiontime as attribute then replacing updated_since timestamp value with lastexecutiontime attribute value.

Configs Screenshots:-

44390-update.png

Here is what i tried:-

I have generate flowfile processor with 2 attributes

lastexecutiontime

2017-12-11T12:00:00Z

url

https://api.someurl.com/api/v2/clients?updated_since=2017-12-11T14:23:52Z∾cess_token=xxxxxxxxxxxxxxx...

Configs:-

44391-generateflowfile.png

Then i used update attribute to update the url with last executiontime after updated_since

UpdateAttribute Configs:-

44392-update.png

Output Flowfile:-

44393-output.png

As you can see above after updated_since having lastexecutiontime value.

Re: Add last execution time with getHTTP processor to REST API URL as updated since parameter

New Contributor

Hi Shu,

thank you for your help. But I got some last questions. Where does the attribute lastexecutiontime come from? I don't find a solution to access the timestamp of the last execution of a processor. And which processor do you use for an API call? It seems that it is not allowed to have an input stream for the getHttp processor. So i can not have a updateAttributes processor before the getHttp.

Thanks a lot
Michael

Re: Add last execution time with getHTTP processor to REST API URL as updated since parameter

Super Guru
@Michael Hannekum

You can use InvokeHTTP processor instead of Get HTTP, InvokeHTTP accepts input stream we need to have Generate Flowfile processor to trigger the flow.

HTTP MethodGETHTTP request method (GET, POST, PUT, DELETE, HEAD, OPTIONS). Arbitrary methods are also supported. Methods other than POST and PUT will be sent without a message body.
Supports Expression Language: true

Results from invoke http processor will have Date attribute associated with it get that date attribute value and store it in hbase, hive, distribute cache server and pull that value every time before invoke http processor by using appropriate processor to pull fetchhbase,selecthiveql,fetchdistributedmapcache processors and this value would be your lastexecutiontime value. Then use update attribute processor to update your URL after updated_since.

Flow:-

GenerateFlowFile //add your url in this processor,schedule this processor as cron or timer driven
Fetch lastexecutiontime //after fetching prepare attribute
Update Attribute //to update updated_Since value
Invoke HTTP //to pull results
Store the lastexecutiontime //store the date attribute value to pull again in your 2 fetch step.

Re: Add last execution time with getHTTP processor to REST API URL as updated since parameter

New Contributor

Thank you Shu for your answer. I will think about this workflow.

I noticed a start and end date in my status history for my getHttp processor. Is it maybe possible to get access to this variable? So I can use the start date as updated_since.

45399-bildschirmfoto-2017-12-13-um-103538.png

Re: Add last execution time with getHTTP processor to REST API URL as updated since parameter

Super Guru
@Michael Hannekum

We don't have Start,End attributes associated with the result flowfile and we cannot create them in GetHTTP processor it self. Think about the suggested flow..!!

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors.

Don't have an account?
Coming from Hortonworks? Activate your account here