Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Read files in s3 bucket one by one using fetch s3 without using lists3 in nifi

avatar
Explorer

Hi 

 

I am trying to read files from s3 bucket using nifi and i dont want to use lists3 as it doesn't accept any input.  I want to start the flow based on a condition. Like there are few processors to be executed before it comes to read the s3 files using fetch s3 . When i add list s3 it doesn't accept any input.

Is there any solution for this.

 

Thanks

Sunil B

3 REPLIES 3

avatar

Though I have not tried it, I suspect this to be possible.

 

ListS3 typically provides the input of FetchS3, if you want to imitate this, consider manually running ListS3 and carefully inspecting the flowfiles it creates. (Content and attributes/metadata).

 

From here you can probably simulate the flowfile e.g. with generateflowfile to test if you can use FetchS3 without ListS3. 

 

If this succeeds, you can update your flow to make sure it provides the right inputs.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.

avatar
Explorer
Hi Dennis,

Thanks for the reply.

 

I tried but fetch s3 is unable to list the file like it is unable to read the filename.

 

Thanks

Sunil B

avatar
New Contributor

Fetch needs a filename to be able to fetch the file.  If known you can set it as an attribute on the file or a parameter context to pass in as the filename. If you don't know what it will be then you'll need to use a list processor.