Nifi vs Kafka and ESB

[#1]

Hi,

I am very new to Nifi and HDF and hence finding it tough to understand the USP of Nifi with respect to other data transport mechanisms. So any help would be grateful.

• Is NiFi’s primary interaction only through UI?
• How different is Nifi from Kafka or any enterprise ESB apart from the visual data flow aspect?
• Especially when comparing with Kafka, what is common between them and where does they differ?

My understanding of the Nifi features with respect to Kafka

• Visual Command and Control - Not available in Kafka
• Data lineage - Something that can be done with Apache Atlas for Kafka?
• Data prioritisation - I presume this can be controlled with a combination of topics and consumers and consumer groups in Kafka
• Back pressure - As Kafka can retain data, consumers can always replay the data and catch-up
• Control Latency vs Throughput - Similar to back pressure and prioritisation, this can be controlled with consumers and topics with data retention
• Security - Kafka also has got security implementation
• Scaling - Build a Kafka cluster

[#2 · Accepted solution]

@bigdata.neophyte Here are a few answers for you:

• NiFi can be interacted with via the UI as well as its REST API. The API is documented at https://nifi.apache.org/docs/nifi-docs/rest-api/index.html.
• NiFi is primarily a data flow tool whereas Kafka is a broker for a pub/sub type of use pattern. Kafka is frequently used as the backing mechanism for NiFi flows in a pub/sub architecture, so while they work well together they provide two different functions in a given solution.
• NiFi does have a visual command and control mechanism, while Kafka does not have a native command and control GUI
• Apache Atlas, Kafka, and NiFi all can work together to provide a comprehensive lineage / governance solution. There is a high level architecture slide at https://hortonworks.com/apache/atlas/#section_2 as well as a tutorial that might help this make more sense at https://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/.
• Data prioritization, back pressure, and balancing latency and throughput are all within NiFi's many strong points and these can be leveraged easily. Kafka does really not provide data prioritization.
• Security aspects of both Kafka and NiFi are tightly integrated with Apache Ranger, take a look at https://hortonworks.com/apache/ranger/ for additional details.

Hope this helps, and please accept the answer if this was helpful.