NiFi: TailFile on Remote Hosts

[#1]

Can NiFi be configured to tail logs on remote hosts without having to deploy NiFi on hosts outside of processing cluster?

Current work around is to update application to use log4 UdpAppender and publish events on UDP and then use UDP listener.

[#2 · Accepted solution]

Currently TailFile only works on a systems local system. In order to coordinate receiving data from a remote host there is numerous different options and configurations that need to be taken into account. Tail File was designed to fulfill a targeted goal, tail a local file. This is done to create a smaller building block which can build up the bigger system and is easier to re-use for multiple different applications.

Getting log file data off a remote host with all the perks of NiFi is something we are working toward with a sub-project tentatively called "MiNiFi". There is a design proposal requesting feedback here: "http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Proposal-for-an-Apache-NiFi-sub-project-MiNiFi-td6141.html". Please reply with any comments or suggestions!

[#3]

You can run a Nifi instance locally in a small JVM to gather the log file data and send it to a larger cluster for processing. I would use ListenSyslog processor rather than UDP and point Log4J there.