Our latest use case includes real time ingestion of messages:
NiFi (standalone 1 node) & Kafka (4 brokers) are available in our cluster, however by design and due to security limitations Kafka is restricted from Internet access.
I would like some ideas/tips about what approach is the best. Since we cannot send the data directly to Kafka, NiFi could act as intermediary and publish the data to Kafka topics with the built in processors. This could be done by using the ListenHTTP processor (or any other Listening processor that supports protocols supported by the ESB adaptor), but i read in the docs that only POST operations are allowed there and i was wondering if there are other limitations that restrict listening functionality to the very basics and thus would not be suitable for our use case.
Have you seen use cases where NiFi is used in such a manner, or would you suggest another approach?
I would suggest using the HandleHTTPRequest and HandleHTTPResponse processors instead of ListenHTTP. They provide much more flexibility.
Here is a nice guide from @Chris Gambino: https://community.hortonworks.com/articles/55080/create-a-restful-for-nifi-walmart-case-study.html