question Re: NiFi: TailFile on Remote Hosts in Archives of Support Questions (Read Only)

NiFi: TailFile on Remote Hosts

michael_perez — Mon, 18 Jan 2016 23:41:26 GMT

Can NiFi be configured to tail logs on remote hosts without having to deploy NiFi on hosts outside of processing cluster?

Current work around is to update application to use log4 UdpAppender and publish events on UDP and then use UDP listener.

Re: NiFi: TailFile on Remote Hosts

jpercivall — Tue, 19 Jan 2016 00:06:18 GMT

Currently TailFile only works on a systems local system. In order to coordinate receiving data from a remote host there is numerous different options and configurations that need to be taken into account. Tail File was designed to fulfill a targeted goal, tail a local file. This is done to create a smaller building block which can build up the bigger system and is easier to re-use for multiple different applications.

Getting log file data off a remote host with all the perks of NiFi is something we are working toward with a sub-project tentatively called "MiNiFi". There is a design proposal requesting feedback here: "http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Proposal-for-an-Apache-NiFi-sub-project-MiNiFi-td6141.html". Please reply with any comments or suggestions!

Re: NiFi: TailFile on Remote Hosts

vvaks — Mon, 28 Mar 2016 07:00:16 GMT

You can run a Nifi instance locally in a small JVM to gather the log file data and send it to a larger cluster for processing. I would use ListenSyslog processor rather than UDP and point Log4J there.