Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi vs Kafka and ESB

Solved Go to solution

Nifi vs Kafka and ESB

New Contributor

Hi,

I am very new to Nifi and HDF and hence finding it tough to understand the USP of Nifi with respect to other data transport mechanisms. So any help would be grateful.

  • Is NiFi’s primary interaction only through UI?
  • How different is Nifi from Kafka or any enterprise ESB apart from the visual data flow aspect?
  • Especially when comparing with Kafka, what is common between them and where does they differ?

My understanding of the Nifi features with respect to Kafka

  • Visual Command and Control - Not available in Kafka
  • Data lineage - Something that can be done with Apache Atlas for Kafka?
  • Data prioritisation - I presume this can be controlled with a combination of topics and consumers and consumer groups in Kafka
  • Back pressure - As Kafka can retain data, consumers can always replay the data and catch-up
  • Control Latency vs Throughput - Similar to back pressure and prioritisation, this can be controlled with consumers and topics with data retention
  • Security - Kafka also has got security implementation
  • Scaling - Build a Kafka cluster
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Nifi vs Kafka and ESB

Rising Star

@bigdata.neophyte Here are a few answers for you:

  • NiFi can be interacted with via the UI as well as its REST API. The API is documented at https://nifi.apache.org/docs/nifi-docs/rest-api/index.html.
  • NiFi is primarily a data flow tool whereas Kafka is a broker for a pub/sub type of use pattern. Kafka is frequently used as the backing mechanism for NiFi flows in a pub/sub architecture, so while they work well together they provide two different functions in a given solution.
  • NiFi does have a visual command and control mechanism, while Kafka does not have a native command and control GUI
  • Apache Atlas, Kafka, and NiFi all can work together to provide a comprehensive lineage / governance solution. There is a high level architecture slide at https://hortonworks.com/apache/atlas/#section_2 as well as a tutorial that might help this make more sense at https://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/.
  • Data prioritization, back pressure, and balancing latency and throughput are all within NiFi's many strong points and these can be leveraged easily. Kafka does really not provide data prioritization.
  • Security aspects of both Kafka and NiFi are tightly integrated with Apache Ranger, take a look at https://hortonworks.com/apache/ranger/ for additional details.

Hope this helps, and please accept the answer if this was helpful.

1 REPLY 1
Highlighted

Re: Nifi vs Kafka and ESB

Rising Star

@bigdata.neophyte Here are a few answers for you:

  • NiFi can be interacted with via the UI as well as its REST API. The API is documented at https://nifi.apache.org/docs/nifi-docs/rest-api/index.html.
  • NiFi is primarily a data flow tool whereas Kafka is a broker for a pub/sub type of use pattern. Kafka is frequently used as the backing mechanism for NiFi flows in a pub/sub architecture, so while they work well together they provide two different functions in a given solution.
  • NiFi does have a visual command and control mechanism, while Kafka does not have a native command and control GUI
  • Apache Atlas, Kafka, and NiFi all can work together to provide a comprehensive lineage / governance solution. There is a high level architecture slide at https://hortonworks.com/apache/atlas/#section_2 as well as a tutorial that might help this make more sense at https://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/.
  • Data prioritization, back pressure, and balancing latency and throughput are all within NiFi's many strong points and these can be leveraged easily. Kafka does really not provide data prioritization.
  • Security aspects of both Kafka and NiFi are tightly integrated with Apache Ranger, take a look at https://hortonworks.com/apache/ranger/ for additional details.

Hope this helps, and please accept the answer if this was helpful.