Hi, i want to ask why it recommanded to use apache nifi and kafka and storm toghether? we can't use storm with nifi only? what is the benefits to add kafka in the process? thanks,HI, i want to know why it recomanded to work with apache nifi and kafka and storm toghethert? we can't use strom and nifi directly? what are the benefits to use kafka like coordinator? thanks
It is not mandatory to use any of these components together (Nifi, Storm, Kafka). They just work well together to provide an end-to-end stream processing platform. Having said that, clients will wind up using only the components that make sense for their particular use case. Below is a brief explanation at where each of these tools excels.
Kafka is a low latency message broker that follows the publish-subscribe model. It is similar in utility to a JMS broker. Kafka is good at providing the modularity between the message/event producers and consumers without either of them waiting for the other to complete. Hence, a message producer can produce messages/events as fast as it needs, dump them into Kafka and the consumer on the other side can pick up those messages and process them at it's own rate.
So, Kafka can be used when you want to seperate/modularize the relation between data consumers and producers. These consumers/producers can be any system and not necessarily Nifi or Storm.
Nifi is a tool that automates data flow between systems. It operates on both, batch as well as streaming data. It is also capable of doing simple transformations on the data as it flows through such as encryption, data cleansing, data routing, etc...
Nifi can connect to a variety of data sources/destinations including (but not necessarily) Storm and Kafka
Storm is a stream processing/analytics engine that allows you to operate on data-in-flight. It also differs from Nifi in that it allows you to do more complex processing, transformations and analytics than is possible with Nifi. One such example is sliding window analysis. So, essentially, Storm is a complex event processing platform and Nifi is a simple event processing and data flow platform.