Pull logs from a remote server to Nifi

[#1]

I am a beginner in NIFI. And i would like to read live logs from a remote server to a Nifi flow on another system. I have no access to install Nifi on that remote server, please what is the best method to implement the above scenario.
Thanking you so much for your reply.

[#2 · Accepted solution]

@rafy 

There are two commonly used methods used to get actively being written to logs from a source server.

1. Install a MiNiFi agent on the server that utilizes a tailFile processor that is configured to read the log file being produced and then sends those FlowFiles to your NiFi cluster for further processing.  However, as you said, you can not install new services/software on this server, so that rules out this option.

2. Another option is to modify the logger on your source system so that in addition to it logging locally, it also sends log output to an external syslog server.  IN this case that syslog server would be your NiFi cluster with a dataflow that uses the ListenSyslog NiFi processor.  

There is no way for NiFi to connect to a remote server and incrementally pull new lines from a file that is continuously being written to. Now if your source server rolls the logs, It is possible you could have your NiFi use the ListSFTP and FetchSFTP processors to consume those rolled logs.  Downside here is this would no be real time processing of those logs since you are only consuming based on the log rotation configuration on the target server.  And you cannot use these processors to consume the actively being written to log.  Doing so means that NiFi would pull the entire contents of the log each time the processor executes rather than just the newest log lines. 

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

[#3]

Thank you so much for this insight. It really helps &  i really appreciate. And if you have any links that i can use as illustration of the first method i would also be grateful.

[#4]

@rafy 

You'll want to read up on the documentation on the Apache MiNiFi page here:
https://nifi.apache.org/minifi/index.html

Since MiNiFi does not provide a UI from which you can construct a NiFi dataflow, you will need to build the dataflow that you will use on your MiNiFi using a NiFi installation.  

The converter toolkit is what you can  then use to change your NiFi dataflow template into the necessary MiNiFi yaml file.

You may also find these community posts helpful:
https://community.cloudera.com/t5/Support-Questions/How-send-data-from-nifi-to-minifi-same-config/td...
https://community.cloudera.com/t5/Community-Articles/Ingesting-Log-data-using-MiNiFi-NiFi/ta-p/24815...

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt