Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Nifi: how to use folderFilter for fetching files from S3


I have a requirement to access AWS S3 bucket through NIFI and process files into HDFS from the specific subfolder Ex:- S3 bucket name: my_bucket. Folders under my_bucket(S3) have  ABC, BDE,CEF,XGF,BHG,NHY. I have to process files only from BDE & CEF subfolders and ignore others.

I am currently using ListS3 -> Fetch S3 -> Routeonattribute -> UpdateAttribute -> PutHDFS

I am unable to filter folder name on ListS3 or FetchS3. So thought of using Routeonattribute by filtering on absolute.path as below. Could you please help if the above logic is correct?


Super Guru

@Deenag Yes, this is a typical method to filter out flowfiles based on attributes matching expression language.  You setup the routes you want and ignore the rest.