Community Articles
Find and share helpful community-sourced technical articles
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Super Guru

In Apache NiFi 1.2, there are processors to Get and Put data to an MQTT broker, which is popular in IoT because of it's small footprint and speed. MQTT is supported by Eclipse and IBM.

I created an example on the HDP 2.6. I downloaded and installed the latest Apache NiFi 1.2 there as well as an example MQTT Broker (Mosquitto)

To Install Mosquitto on HDP 2.6 (Centos 7.x)

sudo wget cp *.repo /etc/yum.repos.d/
sudo yum -y update
sudo yum -y install mosquitto

To Verify the Settings and Prepare Logs

[root@sandbox opt]# cat /etc/mosquitto/mosquitto.conf

# Place your local configuration in /etc/mosquitto/conf.d/
pid_file /var/run/
persistence true
persistence_location /var/lib/mosquitto/
#log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

[root@sandbox opt]# vi /etc/mosquitto/mosquitto.conf
[root@sandbox opt]# mkdir -p /var/log/mosquitto
[root@sandbox opt]# chmod 777 /var/log/mosquitto/
[root@sandbox opt]# touch /var/log/mosquitto/mosquitto.log
[root@sandbox opt]# chmod 777 /var/log/mosquitto/

Run MQTT Broker Server

mosquitto -d

The default port for MQTT and Mosquitto is 1883. Make sure that port is not blocked by Firewalls, Virus software and if one the sandbox it is exposed.


Running Mosquitto on Sandbox

NiFi PublishMQTT

NiFi ConsumeMQTT

After Running

[root@sandbox demo]# hdfs dfs -ls /mqtt

root hdfs  2783 2016-07-20 14:56 /mqtt/37115929161818
root hdfs  2805 2016-07-20 14:56 /mqtt/37115930927495






Publish MQTT





New Contributor

Great article Tim. I'm thinking a NiFi dataflow to simulate 100's or 1000's IoT sensors would be a great idea.

Super Guru

why simulate. A raspberry pi or two can send thousands of mqtt message a second. you could simulate that with Gatling or JMeter

New Contributor

Hi Can you tell me the PutHDFS Configuration. I'am facing authentication issues.

Also can you provide me the complete steps for this MQTT Demo

Super Guru

kerberos is a tougher, but without kerberos:

set configuration: must have access to hadoop files


set a directory

New Contributor


Great post need to correct this part :

  1. sudo wget
  2. sudo cp *.repo /etc/yum.repos.d/
  3. sudo yum -y update
  4. sudo yum -y install mosquitto

step 1 and 2 are fused.


Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 11:15 AM
Updated by:
Top Kudoed Authors