Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Reading Nested folders using ListS3 Processor

avatar
New Contributor

Hi All,

 

I have to read files from an S3 bucket and put the data in a different bucket.

The bucket name is, say my-bucket

 

The prefixes to read are

customer-sales/year=2022/month=06/day=01/hour=00,

customer-sales/year=2022/month=06/day=01/hour=01

customer-sales/year=2022/month=06/day=01/hour=02... and so on.

I have to read the data for the entire day 01 of month 06.

 

So what prefix should I give in the List S3 processor?

 

I have tried giving just customer-sales/year=2022/month=06/day=01/  assuming that NiFi will automatically read the nested folders but seems that is not happening.

I also tried customer-sales/year=2022/month=06/day=01/.*/ after reading some blogs.

 

Currently, I am using List S3 -> Fetch S3 -> PutS3 processors.

0 REPLIES 0