Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Super Guru

In Apache NiFi 1.2, there are processors to Get and Put data to an MQTT broker, which is popular in IoT because of it's small footprint and speed. MQTT is supported by Eclipse and IBM.

I created an example on the HDP 2.6. I downloaded and installed the latest Apache NiFi 1.2 there as well as an example MQTT Broker (Mosquitto) http://mosquitto.org/.

To Install Mosquitto on HDP 2.6 (Centos 7.x)

sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.reposudo cp *.repo /etc/yum.repos.d/
sudo yum -y update
sudo yum -y install mosquitto

To Verify the Settings and Prepare Logs

[root@sandbox opt]# cat /etc/mosquitto/mosquitto.conf

# Place your local configuration in /etc/mosquitto/conf.d/
pid_file /var/run/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
#log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

[root@sandbox opt]# vi /etc/mosquitto/mosquitto.conf
[root@sandbox opt]# mkdir -p /var/log/mosquitto
[root@sandbox opt]# chmod 777 /var/log/mosquitto/
[root@sandbox opt]# touch /var/log/mosquitto/mosquitto.log
[root@sandbox opt]# chmod 777 /var/log/mosquitto/

Run MQTT Broker Server

mosquitto -d

The default port for MQTT and Mosquitto is 1883. Make sure that port is not blocked by Firewalls, Virus software and if one the sandbox it is exposed.

5922-openport.png

Running Mosquitto on Sandbox

NiFi PublishMQTT

NiFi ConsumeMQTT

After Running

[root@sandbox demo]# hdfs dfs -ls /mqtt

root hdfs  2783 2016-07-20 14:56 /mqtt/37115929161818
root hdfs  2805 2016-07-20 14:56 /mqtt/37115930927495

ConsumeMQTT

5923-consumemqttdiagram.png

5924-consumemqtt.png

5925-consumemqtt2.png

5927-consumemqtt4.png

Publish MQTT

5931-publishtomqtt.png

5929-publishmqtt1.png

5930-publishmqtt2.png

Resources:


publishtomqtt.png
11,915 Views
Comments
New Contributor

Great article Tim. I'm thinking a NiFi dataflow to simulate 100's or 1000's IoT sensors would be a great idea.

Super Guru

why simulate. A raspberry pi or two can send thousands of mqtt message a second. you could simulate that with Gatling or JMeter

New Contributor

Hi Can you tell me the PutHDFS Configuration. I'am facing authentication issues.

Also can you provide me the complete steps for this MQTT Demo

Super Guru

kerberos is a tougher, but without kerberos:

set configuration: must have access to hadoop files

/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

set a directory

New Contributor

Hello,

Great post need to correct this part :

  1. sudo wget http://download.opensuse.org/repositories/home:/oojah:/mqtt/CentOS_CentOS-6/home:oojah:mqtt.repo
  2. sudo cp *.repo /etc/yum.repos.d/
  3. sudo yum -y update
  4. sudo yum -y install mosquitto

step 1 and 2 are fused.

Regards

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 11:15 AM
Updated by:
 
Contributors
Top Kudoed Authors