Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

what is the relation between nifi and storm in HDF?

Solved Go to solution
Highlighted

what is the relation between nifi and storm in HDF?

Contributor

It seems that storm is mentioned rarely in HDF.

Can nifi replace storm?

Nifi and storm is independent to each other?(no processer in nifi communicate with storm )

1 ACCEPTED SOLUTION

Accepted Solutions

Re: what is the relation between nifi and storm in HDF?

Explorer

HDF, as a data-in-motion platform, contains NIFI as one component, for data ingestion & flow management, and Storm as another component for complex event processing.

NIFI and Storm are complementary to each other. NIFI is not yet another processing framework, but to deliver data to those frameworks such as Storm. Both NIFI and Storm can handle processing jobs, but we tend to leverage NiFi for Simple Event Processing, whereas Storm for Complex Event Processing. In terms of Simple Event Processing, usually you are able to operate on a single piece of data by itself (or joining with an Enrichment Dataset). Complex Event Processing typically normally refers to processing data from multiple streams, think of a JOIN operation, or rolling window aggregation type of analytics.

You can embed NIFI-external-modules into Storm for integration purposes, in order to leverage both components to achieve your ultimate streaming analytics goals. As an example, you can either embed a "NiFiSpout" into Storm to pull data from NIFI, or "NiFiBolt" to send data back to NIFI

View solution in original post

3 REPLIES 3
Highlighted

Re: what is the relation between nifi and storm in HDF?

I suggest having a gander at related comments here.

NiFi cannot replace Storm for some use cases. NiFi can replace Storm for some use cases.

NiFi's sweet spot is simple event processing (one event context at a time). Storm, Spark, Flink, etc. are all more powerful when it comes to complex (windowed, or cross-event) processing, but require writing code.

Do you have a particular use case in mind?

Re: what is the relation between nifi and storm in HDF?

Explorer

HDF, as a data-in-motion platform, contains NIFI as one component, for data ingestion & flow management, and Storm as another component for complex event processing.

NIFI and Storm are complementary to each other. NIFI is not yet another processing framework, but to deliver data to those frameworks such as Storm. Both NIFI and Storm can handle processing jobs, but we tend to leverage NiFi for Simple Event Processing, whereas Storm for Complex Event Processing. In terms of Simple Event Processing, usually you are able to operate on a single piece of data by itself (or joining with an Enrichment Dataset). Complex Event Processing typically normally refers to processing data from multiple streams, think of a JOIN operation, or rolling window aggregation type of analytics.

You can embed NIFI-external-modules into Storm for integration purposes, in order to leverage both components to achieve your ultimate streaming analytics goals. As an example, you can either embed a "NiFiSpout" into Storm to pull data from NIFI, or "NiFiBolt" to send data back to NIFI

View solution in original post

Highlighted

Re: what is the relation between nifi and storm in HDF?

Contributor

thank you!

Don't have an account?
Coming from Hortonworks? Activate your account here