Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

API Incremental load in NiFi using end_time

Solved Go to solution

API Incremental load in NiFi using end_time

New Contributor

Hi 

 

I am trying to do an incremental load from API using the API end_time tag. 

 

Trying to use the below syntax in RouteAttribute processor to fetch only records updated till yesterday night - 23:59:59 

Could you tell me what is the exact syntax for checking end_time here? 

${end_time:lt(${now():toNumber():minus(86400000) })}

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: API Incremental load in NiFi using end_time

Master Guru

@rki 

 

Your NiFi Expression Language (EL) statement shared expects that the inbound FlowFile already has FlowFile attribute named "end_time" with some value assigned to it.

 

What does that value look like?  How was it created?

${end_time:lt(${now():toNumber():minus(86400000)})}

 

Let's break down the embedded NiFi EL statement first:

${now():toNumber():minus(86400000)}

The now() function returns the current timestamp.

The toNumber() function converts that timestamp in to milliseconds since midnight GMT Jan 1, 1970.

The minus() function subtracts the number passed to the function (86400000) from the above calculated milliseconds.

 

Assuming that the "end_time" attribute returns some number that represents the number of milliseconds since midnight GMT Jan 1, 1970 also and that number is less than the value calculated by the embedded NiFi EL, the NiFi EL will return "true".  Essential all files were the end_time is older then 24 hours from the current timestamp.

 

The FlowFile would then get routed to the relationship named by your RouteOnAttribute custom property.
If false is returned and no other custom properties match, the FlowFile would be routed to unmatched.

 

If you are really trying to only route FlowFiles were the "end_time" milliseconds falls within the last 24 hours only, then you would want to use the ge() function instead of the lt() function.

 

Hope this helps,

Matt

View solution in original post

2 REPLIES 2
Highlighted

Re: API Incremental load in NiFi using end_time

Master Guru

@rki 

 

Your NiFi Expression Language (EL) statement shared expects that the inbound FlowFile already has FlowFile attribute named "end_time" with some value assigned to it.

 

What does that value look like?  How was it created?

${end_time:lt(${now():toNumber():minus(86400000)})}

 

Let's break down the embedded NiFi EL statement first:

${now():toNumber():minus(86400000)}

The now() function returns the current timestamp.

The toNumber() function converts that timestamp in to milliseconds since midnight GMT Jan 1, 1970.

The minus() function subtracts the number passed to the function (86400000) from the above calculated milliseconds.

 

Assuming that the "end_time" attribute returns some number that represents the number of milliseconds since midnight GMT Jan 1, 1970 also and that number is less than the value calculated by the embedded NiFi EL, the NiFi EL will return "true".  Essential all files were the end_time is older then 24 hours from the current timestamp.

 

The FlowFile would then get routed to the relationship named by your RouteOnAttribute custom property.
If false is returned and no other custom properties match, the FlowFile would be routed to unmatched.

 

If you are really trying to only route FlowFiles were the "end_time" milliseconds falls within the last 24 hours only, then you would want to use the ge() function instead of the lt() function.

 

Hope this helps,

Matt

View solution in original post

Highlighted

Re: API Incremental load in NiFi using end_time

New Contributor

Thanks for the detailed explanation. This helps!

Don't have an account?
Coming from Hortonworks? Activate your account here