@rafy
There are two commonly used methods used to get actively being written to logs from a source server.
1. Install a MiNiFi agent on the server that utilizes a tailFile processor that is configured to read the log file being produced and then sends those FlowFiles to your NiFi cluster for further processing. However, as you said, you can not install new services/software on this server, so that rules out this option.
2. Another option is to modify the logger on your source system so that in addition to it logging locally, it also sends log output to an external syslog server. IN this case that syslog server would be your NiFi cluster with a dataflow that uses the ListenSyslog NiFi processor.
There is no way for NiFi to connect to a remote server and incrementally pull new lines from a file that is continuously being written to. Now if your source server rolls the logs, It is possible you could have your NiFi use the ListSFTP and FetchSFTP processors to consume those rolled logs. Downside here is this would no be real time processing of those logs since you are only consuming based on the log rotation configuration on the target server. And you cannot use these processors to consume the actively being written to log. Doing so means that NiFi would pull the entire contents of the log each time the processor executes rather than just the newest log lines.
If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.
Thank you,
Matt
