Log Forwarding/Ingestion Patterns
Log forwarding & ingestion is a key starting point for many logging initiatives such as log analytics, cyber security, anomaly & bot detection, etc etc. This article will focus few (not comprehensive) patterns for log forwarding/ingestion using NiFi.
Commonly rsyslog is used to capture and ship log messages.“Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IPnetwork. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds features such as using TCP for transport.”
More on how to configure rsyslog: here
NiFi is able to ingest messages from rsyslog over TCP or UDP via ListenSysLog processor. This allows for little to no coding.
Patterns
Pattern A
A minimalist design. Rsyslog is configured to simply forward log messages to a NiFi cluster. Rsyslog /etc/rsyslog.conf file needs to be configured to forward messages to a NiFi port identified in ListenSysLog processor.
Pattern B
A MiNiFi listen socket design. MiNiFi is installed on the server(s) leveraging ListenSysLog processor. This pattern offers end to end data linage along with more rich operational capabilities compared to Pattern A. MiNiFi via ListenSysLog will capture rsyslog messages and ship them to NiFi via S2S (site 2 site). Rsyslog is configured to simply forward log messages to a locally installed MiNiFi instance (localhost:port). Rsyslog /etc/rsyslog.conf file needs to be configured to forward messages to a the local MiNiFi port identified in ListenSysLog processor. This design will provide at least once message delivery guarantee.
Pattern C
A MiNiFi tail file design. MiNiFi is installed on the server(s) leveraging TailFile processor unlike Pattern B using ListenSyslog. Both pattern A and B offer end to end data linage and rich operational capabilities. MiNiFi will capture log messages by tailing a directory of files or a file and ship them to NiFi via S2S (site 2 site). Identify a log file to tail (ie /var/log/messages) or a directory for files, start MiNiFi and the log messages will start flow from the server(s) to NiFi. This design will provide at least once message delivery guarantee.
These are a few but common pattens I have developed & implemented in the field with success. Happy log capturing!
