Sensor Data Analysis using HDP and HDF - Part 4

[#1]

Configuring Minifi

Now we will install Minifi, which once configured, will be setup on the node capturing data which raspberry pi 3 here. Download the Minifi tar file on node. All the downloads are available on https://nifi.apache.org/minifi/download.html. I have downloaded version 0.5

1. Download the tar file on the node and unzip.

1. wget http://apache.mirrors.hoobly.com/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz 
2. tar -xzvf minifi-0.5.0-bin.tar.gz

2. Next, we will create the Minifi flow, which will be created on Nifi WebUI. Login to Nifi UI (the one you installed on HDP cluster)

3. On NiFi's web UI, place an Input Port "From raspberry" and connect it to a MergeContent processor.

4. On the same canvas, create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.

5. Inside the process group, create a Remote Process Group (RPG) and give it the URL of your NiFi instance. Also change the protocol to HTTP

6. Create a ExecuteProcess processor and connect it to the RPG, selecting the "From raspberry" input port created earlier.

* The input port (from raspberry pi) mentioned here should be same as that configured in Nifi workflow. In case you see MiNiFi not able to send data to Nifi input port, try reconfiguring it again.

7. Configure the Execute process and enter the absolute location of script which will execute and capture the data. This script will be placed on raspberry pi or data capturing node.

Critical steps: Now since we have created the workflow on UI, we need to convert it to minify readable for which we will use the below tool.

8. Now create a template by right clicking on the Process group and download it by clicking on upper right corner

9. On raspberry pi node download the Converter ToolKit. It can be downloaded using

wget http://apache.claz.org/nifi/minifi/0.5.0/minifi-toolkit-0.5.0-bin.tar.gz

10. Copy the downloaded template to raspberry pi and run the below command from the tool

./bin/config.sh transform /path/to/template.xml /output/path/config.yml

11. Replace this processed file with config.yml file in minifi directory. (/usr/minifi-0.5.0/conf)

12. Now we will setup the script which will run and capture the data. Since in raspberry pi, the sensor data is fetched by python, I have used a wrapper shell script. You can your own bash script and provide the name with its absolute path in the process (refer to point 7)

# cat /home/pi/get_environment.sh (my bash script which runs python script on raspberry)

#Wrapper script#

out=`python /home/pi/get_environment.py`

echo $out|tr ' ' ','

# cat /home/pi/get_environment.py

from sense_hat import SenseHat

import datetime

sense = SenseHat()

sense.clear()

hostname = "real_rasp"

date = datetime.datetime.now()

year=date.year

month=date.month

day=date.day

hour=date.hour

min=date.minute

sec=date.second

long=77.1025

lat=28.7041

pressure = '%.2f' % sense.get_pressure()

temp = '%.2f' % sense.get_temperature()

print hostname,lat,long,year,month,day,hour,min,sec,temp,pressure

13. Once it is done, it is time to start MiNiFi

14. Validation: Now go to Nifi UI and start the input port processor. If the data is reaching the input port, you will see data in the queue next to it.

15. Once you see data in queue, stop the input port for now.

Sample yaml file attached.

config.txt

Links to series

Part 1, Part 2, Part 3, Part 4, Part 5