How to list limited number of directories under hdfs using hdfslist processor in nifi ?

Cloudearnifi — Mon, 27 Feb 2023 18:19:12 GMT

Question;
We have around 6k tables under Hdfs as directories and when we use hdfslist processor in nifi its listing everything and killing the cluster . How can we select limited number of tables may 100 or 200 instead of listing everything ?

Re: How to list limited number of directories under hdfs using hdfslist processor in nifi ?

ckumar — Tue, 28 Feb 2023 12:07:43 GMT

It seems the same question discussed at https://community.cloudera.com/t5/Support-Questions/Passing-list-of-directories-to-ListHdfs-Processor/td-p/364798

To address the limited number of tables is to limit the number of files in HDFS terms, Listing strategy can only be controlled by File Filter and file-filter-mode on what can be listed.

The Listting processing has a two-step

-->What to list " controlled by the filter "

-->From Whereto list "where in your case looks like sub directory under root has a widespread and number of nested sub dirs are huge where the processor is spending time on recursive searching.

If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Chandan

question Re: How to list limited number of directories under hdfs using hdfslist processor in nifi ? in Support Questions

How to list limited number of directories under hdfs using hdfslist processor in nifi ?

Re: How to list limited number of directories under hdfs using hdfslist processor in nifi ?