Lists3 processor to fetch only one layer of subfolders

Ytch — Thu, 14 Nov 2024 14:20:12 GMT

I have a folder in S3 bucket let us say Test/

Now the Test folder has multiple sub folders and each of these subfolders have multiple files.

Now I want to list only the sub folder names within the Test folder and I don't want the names of contents within these sub folders.

I configured the processor like this:
prefix: Test/
With this configuration, I get all the subfolders along with their contents.

When I give the delimiter as /, then I don't get any result.

Re: Lists3 processor to fetch only one layer of subfolders

MattWho — Thu, 14 Nov 2024 21:16:24 GMT

@Ytch

NiFi is designed to move data and the ListS3 processor is designed to list s3 objects/files and not directories.

On the downstream side, NiFi does not have a processor that just create directories. Many out processors will create a missing directory if content is to be written must be written to a directory path that does not exist on the target. But without content those processor will not create any directories.

I am clear on what yoru use case is here, but based on yoru description it is not something any of the stock processors would do. You would need to write something custom (either your own processor or some custom script you can execute via aNiFi scripting processor).

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

question Re: Lists3 processor to fetch only one layer of subfolders in Support Questions

Lists3 processor to fetch only one layer of subfolders

Re: Lists3 processor to fetch only one layer of subfolders