Skip to main content
Cloudearnifi
Bronze ⭐️⭐️⭐️
Feb 27, 2023 6:13 PM UTC
Question

How to list limited number of directories under hdfs using hdfslist processor in nifi ?

Question;
We have around 6k tables under Hdfs as directories and when we use hdfslist processor in nifi its listing everything and killing the cluster . How can we select limited number of tables may 100 or 200 instead of listing everything ?

1 reply

ckumar
Platinum ⭐️
Platinum ⭐️
Feb 28, 2023 12:07 PM UTC

It seems the same question discussed at   https://community.cloudera.com/t5/Support-Questions/Passing-list-of-directories-to-ListHdfs-Processor/td-p/364798

To address the limited number of tables is to limit the number of files in HDFS terms, Listing strategy can only be controlled by File Filter and file-filter-mode on what can be listed.

The Listting processing has a two-step

-->What to list " controlled by the filter "

-->From Whereto list  "where in your case looks like sub directory under root has a widespread and number of nested sub dirs are huge where the processor is spending time on recursive searching. 

 

If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Chandan 

 

 

Translate