The goal of this article is to ingest log data from multiple servers (running MiNiFi) that push log data to a NiFi cluster. The NiFi cluster will listen for the log data on an input port and route to an HDFS directory (determined by the host name). This article will assume you are using Ambari for NiFi installation/administration.
1) Set the nifi.remote.input.host and nifi.remote.input.socket.port of NiFi cluster
Reason: Allows NiFi cluster to use an Input Port (MiNiFi will push to a Remote Processing Group) * NiFi will listen using the Input Port
a) In Ambari, go to NiFi
b) Choose the Configs tab
c) Choose the Advanced nifi-properties
d) Set the nifi.remote.input.host to your NiFi hostname and nifi.remote.input.socket.port to 10000
e) Restart NiFi using Ambari
2) In NiFi – Create a flow for incoming log data (listening [input port] for MiNiFi data)
Reason: Listen for incoming log data and route the log data to an HDFS directory
a) On the NiFi Flow Canvas, drag-and-drop an Input port (Name your Input Port – I named mine “listen_for_minifi_logs”)
b) Drag-and-drop processor RouteOnAttribute
c) Create a connection between Input Port and RouteOnAttribute
d) Configure RouteOnAttribute - Properties (removing the red caution), adding two properties (one property per server you’re installing MiNiFi on)
I have two servers – rcicak0.field.hortonworks.com and rcicak1.field.hortonworks.com, the incoming log data (flowfile) will contain an attribute called “host_name” where we’ll properly route the flowfile depending on the host_name property
e) Drag-and-drop three putHDFS processors & create a connection using hostname_rcicak0, hostname_rcicak1 and unmatched
f) Each putHDFS processor will have a different HDFS directory – configure properties for each putHDFS processor /tmp/rcicak0/, /tmp/rcicak1/ and /tmp/unmatched
g) Configure the HDFS directory – properties
(adding a core-site.xml and the directory – depending on the connection)
h) Play the processors – at this point, the NiFi flow is ready to receive Log data from MiNiFi
3) Setup MiNiFi on at least one server
Reason: MiNiFi needs to push the log data to a remote processing group and delete the log file
a) Download MiNiFi (from http://hortonworks.com/downloads/ ) on each of the servers (that contain log data)
b) Unzip minifi-0.0.1-bin.zip to a directory
c) Complete step 4 below before continuing to d
d) Running with an account that has the read/write permission to the log data directory (to read the file and delete the file) run *location/minifi-0.0.1/bin/minifi.sh start
4) Using a processor group in NiFi, create a MiNiFi Flow (pushing log data to a remote processing group)
Reason: Push the log data to a remote processing group and delete the log file
a) Create a process group (call the group “minifi_flow”)
b) Go into the process group “minifi_flow"
c) Drag-and-drop the processor GetFile
d) Configure the processor GetFile – Properties (IMPORTANT: Any file matching the file filter’s regular expression under input directory [and the recursive subdirectories when set to true], the file will be deleted once the file is stored in MiNiFi’s content repository
In the example above, the file filter looks for hdfs-audit.log.Archive -> in this case is a date
e) Drag-and-drop the UpdateAttribute processor and create a successful connection between GetFile and UpdateAttribute
f) Configure the UpdateAttribute processor – Properties, setting the host_name attribute adding the nifi expression language getting the hostname
g) Drag-and-drop a remote process group – use the nifi.remote.input.host from above for the URL
Wait for the connection to establish before continuing to h
h) Add a connection between UpdateAttribute and the Remote Process Group – under To Input choose listen_for_minifi_logs
i) Select all processors and relationships to create a template (Download the template’s xml file)
j) Use the minifi-toolkit (https://www.apache.org/dyn/closer.lua?path=/nifi/minifi/0.0.1/minifi-toolkit-0.0.1-bin.zip ) and run the config.sh tool “config.sh transform theminifi_flow_template.xml config.yml” -> which will convert the XML to a YML file that will be read by MiNiFi
k) Copy the config.yml file into the minifi-0.0.1/conf directory on each of the MiNiFi servers (if you already have your MiNiFi agent started, restart the agent)
