Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi: TailFile on Remote Hosts

Solved Go to solution
Highlighted

NiFi: TailFile on Remote Hosts

Can NiFi be configured to tail logs on remote hosts without having to deploy NiFi on hosts outside of processing cluster?

Current work around is to update application to use log4 UdpAppender and publish events on UDP and then use UDP listener.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi: TailFile on Remote Hosts

Currently TailFile only works on a systems local system. In order to coordinate receiving data from a remote host there is numerous different options and configurations that need to be taken into account. Tail File was designed to fulfill a targeted goal, tail a local file. This is done to create a smaller building block which can build up the bigger system and is easier to re-use for multiple different applications.

Getting log file data off a remote host with all the perks of NiFi is something we are working toward with a sub-project tentatively called "MiNiFi". There is a design proposal requesting feedback here: "http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Proposal-for-an-Apache-NiFi-sub-project-MiNiFi-td6141.html". Please reply with any comments or suggestions!

View solution in original post

2 REPLIES 2
Highlighted

Re: NiFi: TailFile on Remote Hosts

Currently TailFile only works on a systems local system. In order to coordinate receiving data from a remote host there is numerous different options and configurations that need to be taken into account. Tail File was designed to fulfill a targeted goal, tail a local file. This is done to create a smaller building block which can build up the bigger system and is easier to re-use for multiple different applications.

Getting log file data off a remote host with all the perks of NiFi is something we are working toward with a sub-project tentatively called "MiNiFi". There is a design proposal requesting feedback here: "http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Proposal-for-an-Apache-NiFi-sub-project-MiNiFi-td6141.html". Please reply with any comments or suggestions!

View solution in original post

Highlighted

Re: NiFi: TailFile on Remote Hosts

Guru

You can run a Nifi instance locally in a small JVM to gather the log file data and send it to a larger cluster for processing. I would use ListenSyslog processor rather than UDP and point Log4J there.

Don't have an account?
Coming from Hortonworks? Activate your account here