Getting Started with MiNiFi

Getting started with MiNiFi

In this tutorial, we will learn how configure MiNiFi to send data to NiFi:

1. Installing HDF
2. Installing MiNiFi
3. Setting up the Flow for NiFi
4. Setting up the Flow for MiNiFi
5. Preparing the flow for MiNiFi
6. Configuring and starting MiNiFi
7. Enjoying the data flow!

References

For a primer on HDF, you can refer to the tutorials here Tutorials and User Guides

Installing HDF

1. If you do not have NiFi installed, please follow the instructions found here

NOTE: The above installation guide is for HDF 1.2.0.1, this is the version that matches Apache MiNiFi 0.0.1. Although HDF 2.0 may work, for this exercise -- it is not recommended at this time.

Installing MiNiFi

Now that you have NiFi up and running it is time to download and install MiNiFi.

1. Open a browser. Download the MiNiFi Binaries from Apache MiNiFi Downloads Page. There are two options: tar.gz a format tailored to Linux and a zip file more compatible with Windows. If you are using a Mac either option is just fine.

   

   Figure 1. MiNiFi download page

   For this tutorial I have downloaded the tar.gz on a Mac as shown above in

2. To install MiNiFi, extract the files from the compressed file to a location in which you want to run the application. I have chosen to install it to /Users/apsaltis/minifi-to-nifi

   The image below show MiNiFi downloaded and installed in this directory:

   

Setting up the Flow for NiFi

NOTE: Before starting NiFi we need to enable Site-to-Site communication. To do that do the following:

• Open <$NIFI_INSTALL_DIR>/conf/nifi.properties in your favorite editor
• Change:
  nifi.remote.input.socket.host=
  nifi.remote.input.socket.port=
  nifi.remote.input.secure=true

  To

• Restart NiFi if it was running

Now that we have NiFi up and running and MiNiFi installed and ready to go, the next thing to do is to create our data flow. To do that we are going to first start with creating the flow in NiFi. Remember if you do not have NiFi running execute the following command:

Now we should be ready to create our flow. To do this do the following:

1. Open a browser and go to: http://\:\/nifi on my machine that url looks is http://127.0.0.1:8080/nifi and going to it in the browser looks like this:

   Figure 2. Empty NiFi Canvas

2. The first thing we are going to do is setup an Input Port. This is the port that MiNiFi will be sending data to. To do this drag the Input Port icon to the canvas and call it "From MiNiFi" as show below in figure 3.

   Figure 3. Adding the Input Port

3. Now that the Input Port is configured we need to have somewhere for the data to go once we receive it. In this case we will keep it very simple and just log the attributes. To do this drag the Processor icon to the canvas and choose the LogAttribute processor as shown below in figure 4.

   Figure 4. Adding the LogAttribute processor

4. Now that we have the input port and the processor to handle our data, we need to connect them. After creating the connection your data flow should look like figure 5 below.

   Figure 5. NiFi Flow

5. We are now ready to build the MiNiFi side of the flow. To do this do the following:
   • Add a GenerateFlowFile processor to the canvas (don't forget to configure the properties on it)
   • Add a Remote Processor Group to the canvas as shown below in Figure 6

   

   Figure 6. Adding the Remote Processor Group

   • For the URL copy and paste the URL for the NiFi UI from your browser
   • Connect the GenerateFlowFile to the Remote Process Group as shown below in figure 7. (You may have to refresh the Remote Processor Group, before the input port will be available)

   

   Figure 7. Adding GenerateFlowFile Connection to Remote Processor Group

6. Your canvas should now look similar to what is shown below in figure 8.

   

   Figure 8. Adding GenerateFlowFile Connection to Remote Processor Group

7. There is one last step we need to take before we can export the template. We need to make sure that we set the back pressure between the GenerateFlowFile processor and the Remote Process Group (RPG). That way if you stop NiFi and not MiNiFi you will not fill-up the hard drive where MiNiFi is running. To set the back pressure do the following:
   1. Right-click on the "From MiNiFi" connection and choose "Configure"
   2. Choose the "Settings" tab
   3. Set the "Back pressure object threshold" and "Back pressure data size threshold" to 10000 and 1 GB respectively.
8. The next step is to generate the flow we need for MiNiFi. To do this do the following steps:
   • Create a template for MiNiFi illustrated below in figure 9. Figure 9. Creating a template
   • Select the GenerateFlowFile and the NiFi Flow Remote Processor Group (these are the only things needed for MiMiFi)
   • Select the "Create Template" button from the toolbar
   • Choose a name for your template
9. We now need to save our template, as illustrated below in figure 10.

   Figure 10. Template button

10. Now we need to download the template as shown below in figure 11

    Figure 11. Saving a template

11. We are now ready to setup MiNiFi. However before doing that we need to convert the template to YAML format which MiNiFi uses. To do this we need to do the following:
    • Navigate to the minifi-toolkit directory (minifi-toolkit-0.0.1)
    • Transform the template that we downloaded using the following command:

      bin/config.sh transform <INPUT_TEMPLATE> <OUTPUT_FILE>

    For example:

    bin/config.sh transform MiNiFi_Flow.xml config.yml

12. Next copy the config.yml to the minifi-0.0.1/conf directory. That is the file that MiNiFi uses to generate the nifi.properties file and the flow.xml.gz for MiNiFi.
13. That is it, we are now ready to start MiNiFi. To start MiNiFi from a command prompt execute the following:
    cd <MINIFI_INSTALL_DIR>
    bin/minifi.sh start

You should be able to now go to your NiFi flow and see data coming in from MiNiFi.