Community Articles

Find and share helpful community-sourced technical articles.
Labels (1)
Super Guru

In Apache NiFi 1.2, there are processors to Get and Put data to an MQTT broker, which is popular in IoT because of it's small footprint and speed. MQTT is supported by Eclipse and IBM.

I created an example on the HDP 2.6. I downloaded and installed the latest Apache NiFi 1.2 there as well as an example MQTT Broker (Mosquitto)

To Install Mosquitto on HDP 2.6 (Centos 7.x)

sudo wget cp *.repo /etc/yum.repos.d/
sudo yum -y update
sudo yum -y install mosquitto

To Verify the Settings and Prepare Logs

[root@sandbox opt]# cat /etc/mosquitto/mosquitto.conf

# Place your local configuration in /etc/mosquitto/conf.d/
pid_file /var/run/
persistence true
persistence_location /var/lib/mosquitto/
#log_dest file /var/log/mosquitto/mosquitto.log
include_dir /etc/mosquitto/conf.d

[root@sandbox opt]# vi /etc/mosquitto/mosquitto.conf
[root@sandbox opt]# mkdir -p /var/log/mosquitto
[root@sandbox opt]# chmod 777 /var/log/mosquitto/
[root@sandbox opt]# touch /var/log/mosquitto/mosquitto.log
[root@sandbox opt]# chmod 777 /var/log/mosquitto/

Run MQTT Broker Server

mosquitto -d

The default port for MQTT and Mosquitto is 1883. Make sure that port is not blocked by Firewalls, Virus software and if one the sandbox it is exposed.


Running Mosquitto on Sandbox

NiFi PublishMQTT

NiFi ConsumeMQTT

After Running

[root@sandbox demo]# hdfs dfs -ls /mqtt

root hdfs  2783 2016-07-20 14:56 /mqtt/37115929161818
root hdfs  2805 2016-07-20 14:56 /mqtt/37115930927495






Publish MQTT






Great article Tim. I'm thinking a NiFi dataflow to simulate 100's or 1000's IoT sensors would be a great idea.

why simulate. A raspberry pi or two can send thousands of mqtt message a second. you could simulate that with Gatling or JMeter

Hi Can you tell me the PutHDFS Configuration. I'am facing authentication issues.

Also can you provide me the complete steps for this MQTT Demo

kerberos is a tougher, but without kerberos:

set configuration: must have access to hadoop files


set a directory


Great post need to correct this part :

  1. sudo wget
  2. sudo cp *.repo /etc/yum.repos.d/
  3. sudo yum -y update
  4. sudo yum -y install mosquitto

step 1 and 2 are fused.


Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎08-17-2019 11:15 AM
Updated by:
Top Kudoed Authors