IoT Example in Apache NiFi: Consuming and Producing MQTT Messages

[#1]

In Apache NiFi 1.2, there are processors to Get and Put data to an MQTT broker, which is popular in IoT because of it's small footprint and speed. MQTT is supported by Eclipse and IBM.

I created an example on the HDP 2.6. I downloaded and installed the latest Apache NiFi 1.2 there as well as an example MQTT Broker (Mosquitto) http://mosquitto.org/.

To Install Mosquitto on HDP 2.6 (Centos 7.x)

sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.reposudo cp *.repo /etc/yum.repos.d/
sudo yum -y update
sudo yum -y install mosquitto

To Verify the Settings and Prepare Logs

[root@sandbox opt]# cat /etc/mosquitto/mosquitto.conf

# Place your local configuration in /etc/mosquitto/conf.d/
pid_file /var/run/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
#log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

[root@sandbox opt]# vi /etc/mosquitto/mosquitto.conf
[root@sandbox opt]# mkdir -p /var/log/mosquitto
[root@sandbox opt]# chmod 777 /var/log/mosquitto/
[root@sandbox opt]# touch /var/log/mosquitto/mosquitto.log
[root@sandbox opt]# chmod 777 /var/log/mosquitto/

Run MQTT Broker Server

mosquitto -d

The default port for MQTT and Mosquitto is 1883. Make sure that port is not blocked by Firewalls, Virus software and if one the sandbox it is exposed.

Running Mosquitto on Sandbox

NiFi PublishMQTT

NiFi ConsumeMQTT

After Running

[root@sandbox demo]# hdfs dfs -ls /mqtt

root hdfs  2783 2016-07-20 14:56 /mqtt/37115929161818
root hdfs  2805 2016-07-20 14:56 /mqtt/37115930927495

ConsumeMQTT

Publish MQTT

Resources:

• http://mosquitto.org/man/mosquitto-8.html
• http://ceit.uq.edu.au/content/mqtt-and-growl
• http://growl.info/
• http://www.eclipse.org/paho/

[#2]

Great article Tim. I'm thinking a NiFi dataflow to simulate 100's or 1000's IoT sensors would be a great idea.

[#3]

why simulate. A raspberry pi or two can send thousands of mqtt message a second. you could simulate that with Gatling or JMeter

[#4]

Hi Can you tell me the PutHDFS Configuration. I'am facing authentication issues.

Also can you provide me the complete steps for this MQTT Demo

[#5]

kerberos is a tougher, but without kerberos:

set configuration: must have access to hadoop files

/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

set a directory

[#6]

Hello,

Great post need to correct this part :

1. sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.repo
2. sudo cp *.repo /etc/yum.repos.d/
3. sudo yum -y update
4. sudo yum -y install mosquitto

step 1 and 2 are fused.

Regards