API Incremental load in NiFi using end_time

rki — Fri, 29 Nov 2019 10:20:26 GMT

I am trying to do an incremental load from API using the API end_time tag.

Trying to use the below syntax in RouteAttribute processor to fetch only records updated till yesterday night - 23:59:59

Could you tell me what is the exact syntax for checking end_time here?

${end_time:lt(${now():toNumber():minus(86400000) })}

Re: API Incremental load in NiFi using end_time

MattWho — Thu, 05 Dec 2019 18:35:30 GMT

@rki

Your NiFi Expression Language (EL) statement shared expects that the inbound FlowFile already has FlowFile attribute named "end_time" with some value assigned to it.

What does that value look like? How was it created?

${end_time:lt(${now():toNumber():minus(86400000)})}

Let's break down the embedded NiFi EL statement first:

${now():toNumber():minus(86400000)}

The now() function returns the current timestamp.

The toNumber() function converts that timestamp in to milliseconds since midnight GMT Jan 1, 1970.

The minus() function subtracts the number passed to the function (86400000) from the above calculated milliseconds.

Assuming that the "end_time" attribute returns some number that represents the number of milliseconds since midnight GMT Jan 1, 1970 also and that number is less than the value calculated by the embedded NiFi EL, the NiFi EL will return "true". Essential all files were the end_time is older then 24 hours from the current timestamp.

The FlowFile would then get routed to the relationship named by your RouteOnAttribute custom property.
If false is returned and no other custom properties match, the FlowFile would be routed to unmatched.

If you are really trying to only route FlowFiles were the "end_time" milliseconds falls within the last 24 hours only, then you would want to use the ge() function instead of the lt() function.

Hope this helps,

Matt

Re: API Incremental load in NiFi using end_time

rki — Tue, 10 Dec 2019 09:19:43 GMT

Thanks for the detailed explanation. This helps!

question Re: API Incremental load in NiFi using end_time in Support Questions

API Incremental load in NiFi using end_time

Re: API Incremental load in NiFi using end_time

Re: API Incremental load in NiFi using end_time