Support Questions

Find answers, ask questions, and share your expertise

NIFI : MultiDirectories for ListHDFS


hi all,

I see that listHDFS processor has Directory parameter support Expression Language

So do you know how set mutli-directories for one processor listHDFS ?

Directory : /tmp/{toto|truc} 



If you set it to /tmp and set Recurse Sub Directories to true, then it will list both.


@Bryan : i know for Recurse value, but my dataflow will delete only files in /tmp/toto and /tmp/truc but not in /tmp/Keep and /tmp/News so I can't set only /tmp

Master Guru

@mayki wogno

Setting /tmp will cause listHDFS to produce a listing of files in all 4 of your directories. Following that listing, use a routeOnAttrinute processor to auto-terminate andy listing that were not from /tmp/toto or /tmp/truc before feeding what FlowFiles are left down the rest of your dataflow.


@Matt: thanks.


Hi again,

In my directory /user/prod/ i got thousand files.

How it is possible to configure listHDFS to list only some directories like /user/prod/201703*

So list only all directory 2017 of March ?



What's the best ways to remove old files in the list of directories?

Need i generate one listHDFS processor by same subpath ?




In the latest code in master, there is an improvement to ListHDFS to add a new property:

public static final PropertyDescriptor FILE_FILTER = new PropertyDescriptor.Builder()
    .name("File Filter")
    .description("Only files whose names match the given regular expression will be picked up")

Does this help?


@Bryan, it comes with next version of NIFI ?

Yes its in the latest code that hasn't been released yet, so it would be in the next version which will likely be Apache NiFi 1.2.


@Bryan Bende, in the same way with routeOnAttribute processor, it is possible to regular expression.

In my dataflow listHDFS send flowfile to routeOnAttribute and i need check all subpath list above to route correctly all files in deleteHDFS.

At this moment for each directory I need to create an attribute to match a subpath of directory, so in my case I have created 35 attributes.

13725-attribute-routeonattribute.jpgattribute-routeonattribute.jpg(62.3 kB)

Can you use the "matches" function in expression language?

It should allow any regular expression: