Created on 09-07-201809:44 PM - edited 08-17-201906:28 AM
Log Forwarding/Ingestion Patterns
Log forwarding & ingestion is a key starting point for many logging initiatives such as log analytics, cyber
security, anomaly & bot detection, etc etc. This article will focus few (not comprehensive) patterns for log
forwarding/ingestion using NiFi.
Commonly rsyslog is used to capture and ship log messages.“Rsyslog is an open-source software utility used
on UNIX and Unix-like computer systems for forwarding log messages in an IPnetwork. It implements the
basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration
options and adds features such as using TCP for transport.”
NiFi is able to ingest messages from rsyslog over TCP or UDP via ListenSysLog processor. This allows for
little to no coding.
A minimalist design. Rsyslog is configured to simply forward log messages to a NiFi cluster. Rsyslog
/etc/rsyslog.conf file needs to be configured to forward messages to a NiFi port identified in ListenSysLog
A MiNiFi listen socket design. MiNiFi is installed on the server(s) leveraging ListenSysLog processor. This
pattern offers end to end data linage along with more rich operational capabilities compared to Pattern A.
MiNiFi via ListenSysLog will capture rsyslog messages and ship them to NiFi via S2S (site 2 site). Rsyslog
is configured to simply forward log messages to a locally installed MiNiFi instance (localhost:port). Rsyslog
/etc/rsyslog.conf file needs to be configured to forward messages to a the local MiNiFi port identified in
ListenSysLog processor. This design will provide at least once message delivery guarantee.
A MiNiFi tail file design. MiNiFi is installed on the server(s) leveraging TailFile processor unlike Pattern B
using ListenSyslog. Both pattern A and B offer end to end data linage and rich operational capabilities.
MiNiFi will capture log messages by tailing a directory of files or a file and ship them to NiFi via S2S (site 2
site). Identify a log file to tail (ie /var/log/messages) or a directory for files, start MiNiFi and the log
messages will start flow from the server(s) to NiFi. This design will provide at least once message delivery guarantee.
These are a few but common pattens I have developed & implemented in the field with success. Happy log