Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to make Spooling Directory Source work with sub directories too in Apache Flume

Solved Go to solution

How to make Spooling Directory Source work with sub directories too in Apache Flume

New Contributor

How to make Flume Spooling Directory Source work with the Sub Directories of a folder too. My source folder have several other folders too, I want my flume agent to look into these sub directories too for a file to dump it into sink. Is there any way to do it?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to make Spooling Directory Source work with sub directories too in Apache Flume

New Contributor

There is a patch available FLume-1899, if any body wants he/she can use that for Recursive Directory Search.

2 REPLIES 2

Re: How to make Spooling Directory Source work with sub directories too in Apache Flume

Rising Star

You can create multiple agent sources that point to different sub-directories on the server I believe.  Check out the Flume User Guide and search for "Adding multiple flows in an agent" to see if that helps?  There's likely even more applicable information for you beyond that section as well.

 

It also sounds like you may be wanting dynamic recursive directory searching for Flume... I don't know if that's available right now (I don't think it is), but I would be interested if someone knows one way or the other. :)

Highlighted

Re: How to make Spooling Directory Source work with sub directories too in Apache Flume

New Contributor

There is a patch available FLume-1899, if any body wants he/she can use that for Recursive Directory Search.

Don't have an account?
Coming from Hortonworks? Activate your account here