question Re: NiFi ListS3 Maximum Object Age in Support Questions

NiFi ListS3 Maximum Object Age

acasta — Thu, 20 Apr 2023 14:59:33 GMT

I need to List and Fetch specific files on S3 from April 2022 to December 2022. I was hoping to narrow the listing using the Minimum and Maximum Object Age properties as specificed in the processor documentation.

However, I can;t find the Maximum Object Age property?

Anybody knows how to use it?

Any other strategy that might be useful?

Listing all the files in the prefix takes forever as it contains a huge amount of data.

Thanks.

Re: NiFi ListS3 Maximum Object Age

acasta — Thu, 20 Apr 2023 15:07:28 GMT

From other topics, I saw you are NiFi experts and took the liberty to tag you into this. I apologize ahead if it is against the rules

@MattWho @SAMSAL @steven-matison @DigitalPlumber @cotopaul . Thank you for your help!

Re: NiFi ListS3 Maximum Object Age

cotopaul — Thu, 20 Apr 2023 17:27:30 GMT

hi @acasta,

What NiFi version do you have? Using 1.19.1, within List S3, the Maximum Object Age is present and is in the 8th Property from top to bottom. However, when using 1.15.3, that property is not present at all. So the reason for you not seeing it is most likely your NiFi Version.

1.19.1:

1.15.3

From my point of view, you have two options here:

You either upgrade your NiFi Instance to a newer version.
If you have something in your Tags that you could use you to filter your data, you could set Write Object Tags to True and afterwards use an RouteOnAttribute Processor to filter out your data.

Otherwise, I strongly advise you to wait for a better feedback as I have almost no experience with AWS 😞

Re: NiFi ListS3 Maximum Object Age

acasta — Thu, 20 Apr 2023 17:40:17 GMT

@cotopaul I'm using version 1.18.0, so I guess I'd need to update to a newer version.

Re: NiFi ListS3 Maximum Object Age

DianaTorres — Tue, 02 May 2023 17:29:14 GMT

@acasta Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks