Support Questions

Find answers, ask questions, and share your expertise

Reading Nested folders using ListS3 Processor

avatar
New Contributor

Hi All,

 

I have to read files from an S3 bucket and put the data in a different bucket.

The bucket name is, say my-bucket

 

The prefixes to read are

customer-sales/year=2022/month=06/day=01/hour=00,

customer-sales/year=2022/month=06/day=01/hour=01

customer-sales/year=2022/month=06/day=01/hour=02... and so on.

I have to read the data for the entire day 01 of month 06.

 

So what prefix should I give in the List S3 processor?

 

I have tried giving just customer-sales/year=2022/month=06/day=01/  assuming that NiFi will automatically read the nested folders but seems that is not happening.

I also tried customer-sales/year=2022/month=06/day=01/.*/ after reading some blogs.

 

Currently, I am using List S3 -> Fetch S3 -> PutS3 processors.

0 REPLIES 0