Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Nifi and Storm withour KAfka

New Contributor

Hi, i want to ask why it recommanded to use apache nifi and kafka and storm toghether? we can't use storm with nifi only? what is the benefits to add kafka in the process? thanks,HI, i want to know why it recomanded to work with apache nifi and kafka and storm toghethert? we can't use strom and nifi directly? what are the benefits to use kafka like coordinator? thanks

1 REPLY 1

It is not mandatory to use any of these components together (Nifi, Storm, Kafka). They just work well together to provide an end-to-end stream processing platform. Having said that, clients will wind up using only the components that make sense for their particular use case. Below is a brief explanation at where each of these tools excels.

Kafka

Kafka is a low latency message broker that follows the publish-subscribe model. It is similar in utility to a JMS broker. Kafka is good at providing the modularity between the message/event producers and consumers without either of them waiting for the other to complete. Hence, a message producer can produce messages/events as fast as it needs, dump them into Kafka and the consumer on the other side can pick up those messages and process them at it's own rate.

So, Kafka can be used when you want to seperate/modularize the relation between data consumers and producers. These consumers/producers can be any system and not necessarily Nifi or Storm.

You can read more about Kafka in the below link

https://kafka.apache.org/intro

Nifi

Nifi is a tool that automates data flow between systems. It operates on both, batch as well as streaming data. It is also capable of doing simple transformations on the data as it flows through such as encryption, data cleansing, data routing, etc...

Nifi can connect to a variety of data sources/destinations including (but not necessarily) Storm and Kafka

You can read more about Nifi in the below link

https://nifi.apache.org/docs/nifi-docs/html/overview.html

Storm

Storm is a stream processing/analytics engine that allows you to operate on data-in-flight. It also differs from Nifi in that it allows you to do more complex processing, transformations and analytics than is possible with Nifi. One such example is sliding window analysis. So, essentially, Storm is a complex event processing platform and Nifi is a simple event processing and data flow platform.

You can read more about Storm in the below link

http://storm.apache.org/

So, back to your question. All the above tools are absolutely independent but can also be used together.

- Use Kafka if you need a message broker

- Use Nifi when you want to ingest data and do some simple event processing

- Use Storm when you need to do streaming analytics and complex event processing

- Use a combination of the three when it makes sense (e.g. Kafka+Nifi, Kafka+Storm, Nifi+Storm, Kafka+Nifi+Storm)

I hope this clarifies things.

As always, if you find this post helpful, don't forget to "accept" the answer