Store and compare timestamp in Nifi

[#1]

I have a processor group where I read page ID's from a file and retrieve all posts and comments for these pages, I run this every 2 hours. In the response body I get the timestamp of the created time of the post, I need to record the last largest timestamp of all the posts I received and pull only posts .

One approach I though of was to record the timestamp attribute of each flowfile in a file and compare the value of this file with the next flowfiles timestamp, will this work, if not how could I achieve this.

[#2]

Hi @Ranith Ranawaka

You can use the state and conditions features of the UpdateAttribute processor to implement this

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

[#3]

Thanks, I have the timestamp value in each flow file, I need to compare the timestamp of the current flow files with previous flow files and if it is greater write that to a file using the Put file processor. I tried this with the update attribute processor's advanced features in the following way:

Added a rule named max_timestamp with the condition ${getStateValue("maxTimestamp"):lt(${timestamp})} and set the action for the attribute maxTimestamp as ${timestamp}

I also turned set the store state feature to store locally in the processor, but I do not see the maxTimestamp attribute being set to the flowfiles output from the update attribute processor.

[#4]

Hi @Ranith Ranawaka

In your screenshot you are using 'equals' as a condition rather than 'lt'. Make sure to use the right relation.

Also, you need to add the attribute maxTimestamp to the flowfile.