question Re: ListFile on a huge directory in Support Questions

question Re: ListFile on a huge directory in Support Questions https://community.cloudera.com/t5/Support-Questions/ListFile-on-a-huge-directory/m-p/296894#M218480 Hi, I feel ListS3 should work well even with massive data buckets. It collects only metadata and create a flow file. Below few things you can try and see :1) If you have multiple subdirectories inside that bucket, try to put filter for a specific directory in ListS3 processor (if this helps then you can run multiple processors in parallel pointing to same bucket but different specific sub directory) In the screenshot attached, "2019/Q1" is the one specific directory in side bucket and ListS3 will list only the files of that specific directory. <img src="https://community.cloudera.com/t5/image/serverpage/image-id/27698i973044350C535218/image-size/large?v=v2&px=999" role="button" title="Screenshot 2020-05-29 at 9.02.58 PM.png" alt="Screenshot 2020-05-29 at 9.02.58 PM.png" /><img src="https://community.cloudera.com/t5/image/serverpage/image-id/27699i7B13BB63A53A4D11/image-size/large?v=v2&px=999" role="button" title="Screenshot 2020-05-29 at 9.03.07 PM.png" alt="Screenshot 2020-05-29 at 9.03.07 PM.png" /> 2)Duplicate(copy and paste) the ListS3 processor (which you are trying to run), and run the newly created (duplicated) processor again. As ListS3 tracks(internally) the files already listed, it will not re-list even if you stop and start, so easy way is to duplicate the processor and run that new processor and see. ThanksMahendra Fri, 29 May 2020 15:50:07 GMT hegdemahendra 2020-05-29T15:50:07Z