Support Questions
Find answers, ask questions, and share your expertise

NiFi: TailFile on Remote Hosts

Can NiFi be configured to tail logs on remote hosts without having to deploy NiFi on hosts outside of processing cluster?

Current work around is to update application to use log4 UdpAppender and publish events on UDP and then use UDP listener.

1 ACCEPTED SOLUTION

Currently TailFile only works on a systems local system. In order to coordinate receiving data from a remote host there is numerous different options and configurations that need to be taken into account. Tail File was designed to fulfill a targeted goal, tail a local file. This is done to create a smaller building block which can build up the bigger system and is easier to re-use for multiple different applications.

Getting log file data off a remote host with all the perks of NiFi is something we are working toward with a sub-project tentatively called "MiNiFi". There is a design proposal requesting feedback here: "http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Proposal-for-an-Apache-NiFi-sub-project-MiNiFi-td6141.html". Please reply with any comments or suggestions!

View solution in original post

2 REPLIES 2

Currently TailFile only works on a systems local system. In order to coordinate receiving data from a remote host there is numerous different options and configurations that need to be taken into account. Tail File was designed to fulfill a targeted goal, tail a local file. This is done to create a smaller building block which can build up the bigger system and is easier to re-use for multiple different applications.

Getting log file data off a remote host with all the perks of NiFi is something we are working toward with a sub-project tentatively called "MiNiFi". There is a design proposal requesting feedback here: "http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Proposal-for-an-Apache-NiFi-sub-project-MiNiFi-td6141.html". Please reply with any comments or suggestions!

Guru

You can run a Nifi instance locally in a small JVM to gather the log file data and send it to a larger cluster for processing. I would use ListenSyslog processor rather than UDP and point Log4J there.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.