I just started out investigating Nifi for our use case. We are a company that tracks users for all TV, Internet, Radio etc here in France for our customers. These generate about 5 billion hits a day that we then analyse, aggregate and store. I would love to be able to do this in real time :)
I am currently defining a big data stack; kafka, ignite, hadoop, flink and looking if Nifi could be a good fit as a router. Http(s) get/post requests would be intercepted by Nifi and Nifi would route them based on parameters to the appropriate kafka topics (and reply back with a 200) if all is well. Nifi could also, when required, add/modify parameters in the response back.
Your thoughts would be greatly appreciated.
Hi @Eric Damphousse,
Depending on the http(s) payload size, NiFi could be a good fit for routing the requests.
i would look-into having MiNiFi handling the initial request applying minimal validation / routing attributes, and use Site-To-Site to send to a large(r) cluster of NiFi to connect to Kafka / pulsar etc.