Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

Lists3 processor to fetch only one layer of subfolders

avatar
Explorer

I have a folder in S3 bucket let us say Test/

Now the Test folder has multiple sub folders and each of these subfolders have multiple files.

Now I want to list only the sub folder names within the Test folder and I don't want the names of contents within these sub folders.

I configured the processor like this: 
prefix: Test/
With this configuration, I get all the subfolders along with their contents.

When I give the delimiter as /, then I don't get any result.

1 REPLY 1

avatar
Master Mentor

@Ytch 

NiFi is designed to move data and the ListS3 processor is designed to list s3 objects/files and not directories.  

On the downstream side, NiFi does not have a processor that just create directories.  Many out processors will create a missing directory if content is to be written must be written to a directory path that does not exist on the target.  But without content those processor will not create any directories.

I am clear on what yoru use case is here, but based on yoru description it is not something any of the stock processors would do.  You would need to write something custom (either your own processor or some custom script you can execute via aNiFi scripting processor).

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt