Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Store and compare timestamp in Nifi

avatar
Explorer

I have a processor group where I read page ID's from a file and retrieve all posts and comments for these pages, I run this every 2 hours. In the response body I get the timestamp of the created time of the post, I need to record the last largest timestamp of all the posts I received and pull only posts .

One approach I though of was to record the timestamp attribute of each flowfile in a file and compare the value of this file with the next flowfiles timestamp, will this work, if not how could I achieve this.

3 REPLIES 3

avatar

Hi @Ranith Ranawaka

You can use the state and conditions features of the UpdateAttribute processor to implement this

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

avatar
Explorer

Thanks, I have the timestamp value in each flow file, I need to compare the timestamp of the current flow files with previous flow files and if it is greater write that to a file using the Put file processor. I tried this with the update attribute processor's advanced features in the following way:

Added a rule named max_timestamp with the condition ${getStateValue("maxTimestamp"):lt(${timestamp})} and set the action for the attribute maxTimestamp as ${timestamp}

I also turned set the store state feature to store locally in the processor, but I do not see the maxTimestamp attribute being set to the flowfiles output from the update attribute processor.

45794-update-attribute.jpg

45795-update-attribute-advanced.jpg

avatar

Hi @Ranith Ranawaka

In your screenshot you are using 'equals' as a condition rather than 'lt'. Make sure to use the right relation.

Also, you need to add the attribute maxTimestamp to the flowfile.