If you do not have NiFi installed, please follow the instructions found here
NOTE: The above installation guide is for HDF 188.8.131.52, this is the version that matches Apache MiNiFi 0.0.1. Although HDF 2.0 may work, for this exercise -- it is not recommended at this time.
Now that you have NiFi up and running it is time to download and install MiNiFi.
Open a browser. Download the MiNiFi Binaries from Apache MiNiFi Downloads Page. There are two options: tar.gz a format tailored to Linux and a zip file more compatible with Windows. If you are using a Mac either option is just fine.
Figure 1. MiNiFi download page
For this tutorial I have downloaded the tar.gz on a Mac as shown above in
To install MiNiFi, extract the files from the compressed file to a location in which you want to run the application. I have chosen to install it to /Users/apsaltis/minifi-to-nifi
The image below show MiNiFi downloaded and installed in this directory:
Setting up the Flow for NiFi
NOTE: Before starting NiFi we need to enable Site-to-Site communication. To do that do the following:
Open <$NIFI_INSTALL_DIR>/conf/nifi.properties in your favorite editor
nifi.remote.input.socket.host=localhost <-- This is only being done for this exercise as MiNiFi and NiFi are running on the same host. This is not a recommended way of deploying the two products.
nifi.remote.input.secure=false <-- This implies we are only using HTTP and are not securing the communication between MiNiFi and NiFi. For this exercise that is OK, however, it is important to consider your security needs when deploying these technologies.
Restart NiFi if it was running
Now that we have NiFi up and running and MiNiFi installed and ready to go, the next thing to do is to create our data flow. To do that we are going to first start with creating the flow in NiFi. Remember if you do not have NiFi running execute the following command:
Now we should be ready to create our flow. To do this do the following:
The first thing we are going to do is setup an Input Port. This is the port that MiNiFi will be sending data to. To do this drag the Input Port icon to the canvas and call it "From MiNiFi" as show below in figure 3.
Figure 3. Adding the Input Port
Now that the Input Port is configured we need to have somewhere for the data to go once we receive it. In this case we will keep it very simple and just log the attributes. To do this drag the Processor icon to the canvas and choose the LogAttribute processor as shown below in figure 4.
Figure 4. Adding the LogAttribute processor
Now that we have the input port and the processor to handle our data, we need to connect them. After creating the connection your data flow should look like figure 5 below.
Figure 5. NiFi Flow
We are now ready to build the MiNiFi side of the flow. To do this do the following:
Add a GenerateFlowFile processor to the canvas (don't forget to configure the properties on it)
Add a Remote Processor Group to the canvas as shown below in Figure 6
Figure 6. Adding the Remote Processor Group
For the URL copy and paste the URL for the NiFi UI from your browser
Connect the GenerateFlowFile to the Remote Process Group as shown below in figure 7. (You may have to refresh the Remote Processor Group, before the input port will be available)
Figure 7. Adding GenerateFlowFile Connection to Remote Processor Group
Your canvas should now look similar to what is shown below in figure 8.
Figure 8. Adding GenerateFlowFile Connection to Remote Processor Group
There is one last step we need to take before we can export the template. We need to make sure that we set the back pressure between the GenerateFlowFile processor and the Remote Process Group (RPG). That way if you stop NiFi and not MiNiFi you will not fill-up the hard drive where MiNiFi is running. To set the back pressure do the following:
Right-click on the "From MiNiFi" connection and choose "Configure"
Choose the "Settings" tab
Set the "Back pressure object threshold" and "Back pressure data size threshold" to 10000 and 1 GB respectively.
The next step is to generate the flow we need for MiNiFi. To do this do the following steps:
Create a template for MiNiFi illustrated below in figure 9.
Figure 9. Creating a template
Select the GenerateFlowFile and the NiFi Flow Remote Processor Group (these are the only things needed for MiMiFi)
Select the "Create Template" button from the toolbar
Choose a name for your template
We now need to save our template, as illustrated below in figure 10.
Figure 10. Template button
Now we need to download the template as shown below in figure 11
Figure 11. Saving a template
We are now ready to setup MiNiFi. However before doing that we need to convert the template to YAML format which MiNiFi uses. To do this we need to do the following:
Navigate to the minifi-toolkit directory (minifi-toolkit-0.0.1)
Transform the template that we downloaded using the following command: