- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Apache Nifi and Kafka
- Labels:
-
Apache Kafka
-
Apache NiFi
Created ‎09-22-2016 01:03 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Team,
There is one query regarding Apache Nifi and Kafka. Both are messaging system.
Can someone tell can we replace Nifi with Kafka or vice-versa. And what are advantage of Nifi over Kafka.
Created ‎09-22-2016 01:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Apache NiFi and Apache Kafka are two different tools with different usecases that may slightly overlap. Here is my understanding of the purpose of the two projects.
NiFi is "An easy to use, powerful, and reliable system to process and distribute data."
It is a visual tool (with a REST api) that implements flow-based programming to enable the user to craft flows that will take data from a large variety of different sources, perform enrichment, routing, etc on the data as it's being processed, and output the result to a large variety of destinations. During this process it captures metadata (provenance) on what has happened to each piece of data (FlowFile) as it made its way through the Flow for audit logging and troubleshooting purposes.
"Apache Kafka is publish-subscribe messaging rethought as a distributed commit log"
It is a distributed implementation of the publish-subscribe pattern that allows developers to connect programs to each other in different languages and across a large number of machines. It is more of a building block for distributed computing than it is an all-in-one solution for processing data.
Created ‎09-22-2016 01:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Apache NiFi and Apache Kafka are two different tools with different usecases that may slightly overlap. Here is my understanding of the purpose of the two projects.
NiFi is "An easy to use, powerful, and reliable system to process and distribute data."
It is a visual tool (with a REST api) that implements flow-based programming to enable the user to craft flows that will take data from a large variety of different sources, perform enrichment, routing, etc on the data as it's being processed, and output the result to a large variety of destinations. During this process it captures metadata (provenance) on what has happened to each piece of data (FlowFile) as it made its way through the Flow for audit logging and troubleshooting purposes.
"Apache Kafka is publish-subscribe messaging rethought as a distributed commit log"
It is a distributed implementation of the publish-subscribe pattern that allows developers to connect programs to each other in different languages and across a large number of machines. It is more of a building block for distributed computing than it is an all-in-one solution for processing data.
Created ‎09-22-2016 01:47 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
They work great together.
Using Kafka to distribute once something is ingested and processed in NiFi
http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm/
Kafka is great for connecting NiFi to Storm, Flink, Spark and other processors in hadoop
https://blogs.apache.org/nifi/entry/integrating_apache_nifi_with_apache
