- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Lists3 processor to fetch only one layer of subfolders
- Labels:
-
Apache NiFi
Created ‎11-14-2024 06:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have a folder in S3 bucket let us say Test/
Now the Test folder has multiple sub folders and each of these subfolders have multiple files.
Now I want to list only the sub folder names within the Test folder and I don't want the names of contents within these sub folders.
I configured the processor like this:
prefix: Test/
With this configuration, I get all the subfolders along with their contents.
When I give the delimiter as /, then I don't get any result.
Created ‎11-14-2024 01:16 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Ytch
NiFi is designed to move data and the ListS3 processor is designed to list s3 objects/files and not directories.
On the downstream side, NiFi does not have a processor that just create directories. Many out processors will create a missing directory if content is to be written must be written to a directory path that does not exist on the target. But without content those processor will not create any directories.
I am clear on what yoru use case is here, but based on yoru description it is not something any of the stock processors would do. You would need to write something custom (either your own processor or some custom script you can execute via aNiFi scripting processor).
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
