Reply
New Contributor
Posts: 1
Registered: ‎08-19-2014

In flume, how can i source multiple and dynamic real time files.

I have a program which runs every  minute and uses STRACE command to make multiple log files simultaneously.

For example In first  run - It might start tracing 10 processes with a log file name as unique PIDS, this will keep running till the entire stracing completes. In mean time second run would start, where it might start tracing few more processes ID's.

Now i want to continuosly stream all these STRACE logs to another server using flume. 

I tried using spool but, since i have dynamic multiple real time files so, this was a bad option.

I thougt of using exec (tail) but as described the files names are multiple and quite dynamic so, could not figure out the best way to configure source.

Apprecite suggestions. Thanks.

New Contributor
Posts: 2
Registered: ‎08-21-2014

Re: In flume, how can i source multiple and dynamic real time files.

Have you considered piping the STRACE output through syslogd, and using the Flume syslog source to get the events flowing through your Flume topology in realtime?

Announcements
New solutions