Support Questions

Find answers, ask questions, and share your expertise

How to list limited number of directories under hdfs using hdfslist processor in nifi ?

avatar
New Contributor

Question;
We have around 6k tables under Hdfs as directories and when we use hdfslist processor in nifi its listing everything and killing the cluster . How can we select limited number of tables may 100 or 200 instead of listing everything ?

1 REPLY 1

avatar
Master Collaborator

It seems the same question discussed at   https://community.cloudera.com/t5/Support-Questions/Passing-list-of-directories-to-ListHdfs-Processo...

To address the limited number of tables is to limit the number of files in HDFS terms, Listing strategy can only be controlled by File Filter and file-filter-mode on what can be listed.

The Listting processing has a two-step

-->What to list " controlled by the filter "

-->From Whereto list  "where in your case looks like sub directory under root has a widespread and number of nested sub dirs are huge where the processor is spending time on recursive searching. 

 

If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Chandan