Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Store and compare timestamp in Nifi

Store and compare timestamp in Nifi

New Contributor

I have a processor group where I read page ID's from a file and retrieve all posts and comments for these pages, I run this every 2 hours. In the response body I get the timestamp of the created time of the post, I need to record the last largest timestamp of all the posts I received and pull only posts .

One approach I though of was to record the timestamp attribute of each flowfile in a file and compare the value of this file with the next flowfiles timestamp, will this work, if not how could I achieve this.

3 REPLIES 3

Re: Store and compare timestamp in Nifi

Hi @Ranith Ranawaka

You can use the state and conditions features of the UpdateAttribute processor to implement this

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

Re: Store and compare timestamp in Nifi

New Contributor

Thanks, I have the timestamp value in each flow file, I need to compare the timestamp of the current flow files with previous flow files and if it is greater write that to a file using the Put file processor. I tried this with the update attribute processor's advanced features in the following way:

Added a rule named max_timestamp with the condition ${getStateValue("maxTimestamp"):lt(${timestamp})} and set the action for the attribute maxTimestamp as ${timestamp}

I also turned set the store state feature to store locally in the processor, but I do not see the maxTimestamp attribute being set to the flowfiles output from the update attribute processor.

45794-update-attribute.jpg

45795-update-attribute-advanced.jpg

Re: Store and compare timestamp in Nifi

Hi @Ranith Ranawaka

In your screenshot you are using 'equals' as a condition rather than 'lt'. Make sure to use the right relation.

Also, you need to add the attribute maxTimestamp to the flowfile.

Don't have an account?
Coming from Hortonworks? Activate your account here