Evaluating if Nifi is a good fit for real time processing of 5 billions hits a day

[#1]

All,

I just started out investigating Nifi for our use case. We are a company that tracks users for all TV, Internet, Radio etc here in France for our customers. These generate about 5 billion hits a day that we then analyse, aggregate and store. I would love to be able to do this in real time :)

I am currently defining a big data stack; kafka, ignite, hadoop, flink and looking if Nifi could be a good fit as a router. Http(s) get/post requests would be intercepted by Nifi and Nifi would route them based on parameters to the appropriate kafka topics (and reply back with a 200) if all is well. Nifi could also, when required, add/modify parameters in the response back.

Your thoughts would be greatly appreciated.

[#2]

@Eric Damphousse

Can you contact @Abdelkrim Hadjidj et @Pierre Villard les experts pas loins de vous en france :-)

[#3]

Hi @Eric Damphousse,

Depending on the http(s) payload size, NiFi could be a good fit for routing the requests.

i would look-into having MiNiFi handling the initial request applying minimal validation / routing attributes, and use Site-To-Site to send to a large(r) cluster of NiFi to connect to Kafka / pulsar etc.

the benefit would be that MiNiFi has a small footprint and also allows to capture lineage (data origin) for future track and trace. @Pierre Villard has an excellent blog