Support Questions
Find answers, ask questions, and share your expertise

Setting up Listening NiFi for MQTT & HTTP messages


Setting up Listening NiFi for MQTT & HTTP messages

Rising Star

Our latest use case includes real time ingestion of messages:

  • Sensor data from a Hub/Gateway that is a central point of aggregation for individual sensor devices
  • Events processing from the organization's Enterprise Service Bus

NiFi (standalone 1 node) & Kafka (4 brokers) are available in our cluster, however by design and due to security limitations Kafka is restricted from Internet access.

I would like some ideas/tips about what approach is the best. Since we cannot send the data directly to Kafka, NiFi could act as intermediary and publish the data to Kafka topics with the built in processors. This could be done by using the ListenHTTP processor (or any other Listening processor that supports protocols supported by the ESB adaptor), but i read in the docs that only POST operations are allowed there and i was wondering if there are other limitations that restrict listening functionality to the very basics and thus would not be suitable for our use case.

Have you seen use cases where NiFi is used in such a manner, or would you suggest another approach?

For the sensor data, i guess we could use the ConsumeMQTT (or again ListenHTTP/TCP) depending on what is exactly supported at the Gateway. Thanks in advance! @Timothy Spann @Pierre Villard


Re: Setting up Listening NiFi for MQTT & HTTP messages

Cloudera Employee

I would suggest using the HandleHTTPRequest and HandleHTTPResponse processors instead of ListenHTTP. They provide much more flexibility.

Here is a nice guide from @Chris Gambino: