Our latest use case includes real time ingestion of messages:
Sensor data from a Hub/Gateway that is a central point of aggregation for individual sensor devices
Events processing from the organization's Enterprise Service Bus
NiFi (standalone 1 node) & Kafka (4 brokers) are available in our cluster, however by design and due to security limitations Kafka is restricted from Internet access.
I would like some ideas/tips about what approach is the best. Since we cannot send the data directly to Kafka, NiFi could act as intermediary and publish the data to Kafka topics with the built in processors. This could be done by using the ListenHTTP processor (or any other Listening processor that supports protocols supported by the ESB adaptor), but i read in the docs that only POST operations are allowed there and i was wondering if there are other limitations that restrict listening functionality to the very basics and thus would not be suitable for our use case.
Have you seen use cases where NiFi is used in such a manner, or would you suggest another approach?
For the sensor data, i guess we could use the ConsumeMQTT (or again ListenHTTP/TCP) depending on what is exactly supported at the Gateway. Thanks in advance! @Timothy Spann@Pierre Villard