Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Master Guru

In Apache NiFi 1.2, there are processors to Get and Put data to an MQTT broker, which is popular in IoT because of it's small footprint and speed. MQTT is supported by Eclipse and IBM.

I created an example on the HDP 2.6. I downloaded and installed the latest Apache NiFi 1.2 there as well as an example MQTT Broker (Mosquitto) http://mosquitto.org/.

To Install Mosquitto on HDP 2.6 (Centos 7.x)

sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.reposudo cp *.repo /etc/yum.repos.d/
sudo yum -y update
sudo yum -y install mosquitto

To Verify the Settings and Prepare Logs

[root@sandbox opt]# cat /etc/mosquitto/mosquitto.conf

# Place your local configuration in /etc/mosquitto/conf.d/
pid_file /var/run/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
#log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

[root@sandbox opt]# vi /etc/mosquitto/mosquitto.conf
[root@sandbox opt]# mkdir -p /var/log/mosquitto
[root@sandbox opt]# chmod 777 /var/log/mosquitto/
[root@sandbox opt]# touch /var/log/mosquitto/mosquitto.log
[root@sandbox opt]# chmod 777 /var/log/mosquitto/

Run MQTT Broker Server

mosquitto -d

The default port for MQTT and Mosquitto is 1883. Make sure that port is not blocked by Firewalls, Virus software and if one the sandbox it is exposed.

5922-openport.png

Running Mosquitto on Sandbox

NiFi PublishMQTT

NiFi ConsumeMQTT

After Running

[root@sandbox demo]# hdfs dfs -ls /mqtt

root hdfs  2783 2016-07-20 14:56 /mqtt/37115929161818
root hdfs  2805 2016-07-20 14:56 /mqtt/37115930927495

ConsumeMQTT

5923-consumemqttdiagram.png

5924-consumemqtt.png

5925-consumemqtt2.png

5927-consumemqtt4.png

Publish MQTT

5931-publishtomqtt.png

5929-publishmqtt1.png

5930-publishmqtt2.png

Resources:


publishtomqtt.png
19,251 Views
Comments

Great article Tim. I'm thinking a NiFi dataflow to simulate 100's or 1000's IoT sensors would be a great idea.

why simulate. A raspberry pi or two can send thousands of mqtt message a second. you could simulate that with Gatling or JMeter

Hi Can you tell me the PutHDFS Configuration. I'am facing authentication issues.

Also can you provide me the complete steps for this MQTT Demo

kerberos is a tougher, but without kerberos:

set configuration: must have access to hadoop files

/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

set a directory

Hello,

Great post need to correct this part :

  1. sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.repo
  2. sudo cp *.repo /etc/yum.repos.d/
  3. sudo yum -y update
  4. sudo yum -y install mosquitto

step 1 and 2 are fused.

Regards