Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.
New Contributor

Configuring Minifi


Now we will install Minifi, which once configured, will be setup on the node capturing data which raspberry pi 3 here. Download the Minifi tar file on node. All the downloads are available on https://nifi.apache.org/minifi/download.html. I have downloaded version 0.5

1. Download the tar file on the node and unzip.


1. wget http://apache.mirrors.hoobly.com/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz 
2. tar -xzvf minifi-0.5.0-bin.tar.gz


2. Next, we will create the Minifi flow, which will be created on Nifi WebUI. Login to Nifi UI (the one you installed on HDP cluster)

3. On NiFi's web UI, place an Input Port "From raspberry" and connect it to a MergeContent processor.


4. On the same canvas, create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.

108025-1555650145506.png


5. Inside the process group, create a Remote Process Group (RPG) and give it the URL of your NiFi instance. Also change the protocol to HTTP


6. Create a ExecuteProcess processor and connect it to the RPG, selecting the "From raspberry" input port created earlier.

108026-screen-shot-2019-04-18-at-102550-pm.png

* The input port (from raspberry pi) mentioned here should be same as that configured in Nifi workflow. In case you see MiNiFi not able to send data to Nifi input port, try reconfiguring it again.



7. Configure the Execute process and enter the absolute location of script which will execute and capture the data. This script will be placed on raspberry pi or data capturing node.107996-screen-shot-2019-04-18-at-102624-pm.png

Critical steps: Now since we have created the workflow on UI, we need to convert it to minify readable for which we will use the below tool.


8. Now create a template by right clicking on the Process group and download it by clicking on upper right corner


108033-screen-shot-2019-04-18-at-102802-pm.png

108018-screen-shot-2019-04-18-at-102836-pm.png


9. On raspberry pi node download the Converter ToolKit. It can be downloaded using


wget http://apache.claz.org/nifi/minifi/0.5.0/minifi-toolkit-0.5.0-bin.tar.gz


10. Copy the downloaded template to raspberry pi and run the below command from the tool


./bin/config.sh transform /path/to/template.xml /output/path/config.yml


11. Replace this processed file with config.yml file in minifi directory. (/usr/minifi-0.5.0/conf)

12. Now we will setup the script which will run and capture the data. Since in raspberry pi, the sensor data is fetched by python, I have used a wrapper shell script. You can your own bash script and provide the name with its absolute path in the process (refer to point 7)



# cat /home/pi/get_environment.sh (my bash script which runs python script on raspberry)

#Wrapper script#

out=`python /home/pi/get_environment.py`

echo $out|tr ' ' ','


# cat /home/pi/get_environment.py

from sense_hat import SenseHat

import datetime


sense = SenseHat()

sense.clear()


hostname = "real_rasp"

date = datetime.datetime.now()

year=date.year

month=date.month

day=date.day

hour=date.hour

min=date.minute

sec=date.second

long=77.1025

lat=28.7041

pressure = '%.2f' % sense.get_pressure()

temp = '%.2f' % sense.get_temperature()

print hostname,lat,long,year,month,day,hour,min,sec,temp,pressure


13. Once it is done, it is time to start MiNiFi

108027-screen-shot-2019-04-18-at-103001-pm.png


14. Validation: Now go to Nifi UI and start the input port processor. If the data is reaching the input port, you will see data in the queue next to it.


108054-screen-shot-2019-04-18-at-103035-pm.png

15. Once you see data in queue, stop the input port for now.


Sample yaml file attached.

config.txt


Links to series

Part 1, Part 2, Part 3, Part 4, Part 5

92 Views
0 Kudos
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 02:28 PM
Updated by:
 
Contributors
Top Kudoed Authors