Support Questions

Find answers, ask questions, and share your expertise

Nifi vs Kafka and ESB

Hi,

I am very new to Nifi and HDF and hence finding it tough to understand the USP of Nifi with respect to other data transport mechanisms. So any help would be grateful.

  • Is NiFi’s primary interaction only through UI?
  • How different is Nifi from Kafka or any enterprise ESB apart from the visual data flow aspect?
  • Especially when comparing with Kafka, what is common between them and where does they differ?

My understanding of the Nifi features with respect to Kafka

  • Visual Command and Control - Not available in Kafka
  • Data lineage - Something that can be done with Apache Atlas for Kafka?
  • Data prioritisation - I presume this can be controlled with a combination of topics and consumers and consumer groups in Kafka
  • Back pressure - As Kafka can retain data, consumers can always replay the data and catch-up
  • Control Latency vs Throughput - Similar to back pressure and prioritisation, this can be controlled with consumers and topics with data retention
  • Security - Kafka also has got security implementation
  • Scaling - Build a Kafka cluster
1 ACCEPTED SOLUTION

Rising Star

@bigdata.neophyte Here are a few answers for you:

  • NiFi can be interacted with via the UI as well as its REST API. The API is documented at https://nifi.apache.org/docs/nifi-docs/rest-api/index.html.
  • NiFi is primarily a data flow tool whereas Kafka is a broker for a pub/sub type of use pattern. Kafka is frequently used as the backing mechanism for NiFi flows in a pub/sub architecture, so while they work well together they provide two different functions in a given solution.
  • NiFi does have a visual command and control mechanism, while Kafka does not have a native command and control GUI
  • Apache Atlas, Kafka, and NiFi all can work together to provide a comprehensive lineage / governance solution. There is a high level architecture slide at https://hortonworks.com/apache/atlas/#section_2 as well as a tutorial that might help this make more sense at https://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/.
  • Data prioritization, back pressure, and balancing latency and throughput are all within NiFi's many strong points and these can be leveraged easily. Kafka does really not provide data prioritization.
  • Security aspects of both Kafka and NiFi are tightly integrated with Apache Ranger, take a look at https://hortonworks.com/apache/ranger/ for additional details.

Hope this helps, and please accept the answer if this was helpful.

View solution in original post

1 REPLY 1

Rising Star

@bigdata.neophyte Here are a few answers for you:

  • NiFi can be interacted with via the UI as well as its REST API. The API is documented at https://nifi.apache.org/docs/nifi-docs/rest-api/index.html.
  • NiFi is primarily a data flow tool whereas Kafka is a broker for a pub/sub type of use pattern. Kafka is frequently used as the backing mechanism for NiFi flows in a pub/sub architecture, so while they work well together they provide two different functions in a given solution.
  • NiFi does have a visual command and control mechanism, while Kafka does not have a native command and control GUI
  • Apache Atlas, Kafka, and NiFi all can work together to provide a comprehensive lineage / governance solution. There is a high level architecture slide at https://hortonworks.com/apache/atlas/#section_2 as well as a tutorial that might help this make more sense at https://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/.
  • Data prioritization, back pressure, and balancing latency and throughput are all within NiFi's many strong points and these can be leveraged easily. Kafka does really not provide data prioritization.
  • Security aspects of both Kafka and NiFi are tightly integrated with Apache Ranger, take a look at https://hortonworks.com/apache/ranger/ for additional details.

Hope this helps, and please accept the answer if this was helpful.