Support Questions

Find answers, ask questions, and share your expertise

Nifi: how to use folderFilter for fetching files from S3

avatar
Explorer

I have a requirement to access AWS S3 bucket through NIFI and process files into HDFS from the specific subfolder Ex:- S3 bucket name: my_bucket. Folders under my_bucket(S3) have  ABC, BDE,CEF,XGF,BHG,NHY. I have to process files only from BDE & CEF subfolders and ignore others.

I am currently using ListS3 -> Fetch S3 -> Routeonattribute -> UpdateAttribute -> PutHDFS

I am unable to filter folder name on ListS3 or FetchS3. So thought of using Routeonattribute by filtering on absolute.path as below. Could you please help if the above logic is correct?
${absolute.path:contains('DBE')}

1 REPLY 1

avatar
Super Guru

@Deenag Yes, this is a typical method to filter out flowfiles based on attributes matching expression language.  You setup the routes you want and ignore the rest.