NiFi ListS3 Maximum Object Age

Question

I need to List and Fetch specific files on S3 from April 2022 to December 2022. I was hoping to narrow the listing using the Minimum and Maximum Object Age properties as specificed in the processor documentation.

However, I can;t find the Maximum Object Age property?

Anybody knows how to use it?

Any other strategy that might be useful?

Listing all the files in the prefix takes forever as it contains a huge amount of data.

Thanks.

cotopaul · Accepted Answer

hi @acasta,What NiFi version do you have? Using 1.19.1, within List S3, the Maximum Object Age is present and is in the 8th Property from top to bottom. However, when using 1.15.3, that property is not present at all. So the reason for you not seeing it is most likely your NiFi Version.1.19.1:1.15.3From my point of view, you have two options here:You either upgrade your NiFi Instance to a newer version.If you have something in your Tags that you could use you to filter your data, you could set Write Object Tags to True and afterwards use an RouteOnAttribute Processor to filter out your data.Otherwise, I strongly advise you to wait for a better feedback as I have almost no experience with AWS 😞

acasta · Answer

From other topics, I saw you are NiFi experts and took the liberty to tag you into this. I apologize ahead if it is against the rules

@MattWho @SAMSAL @steven-matison @DigitalPlumber @cotopaul . Thank you for your help!

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.