question What types of data ingestion I can use ? in Archives of Support Questions (Read Only)

What types of data ingestion I can use ?

flyingherointhe — Thu, 13 Dec 2018 07:14:33 GMT

Can I use Kafka without nifi for data ingestion ?

Re: What types of data ingestion I can use ?

Shelton — Sat, 17 Aug 2019 22:48:47 GMT

YES, you can use Kafka connectors to read from or write to external systems, manage data flow, and scale the system—all without writing new code. The most popular use case is ETL pipelines. Kafka is a perfect tool for building data pipelines as it’s reliable, scalable, and efficient.
Check https://www.confluent.io/product/confluent-platform/

Re: What types of data ingestion I can use ?

flyingherointhe — Thu, 13 Dec 2018 07:48:57 GMT

thank you, so I do not need NiFi in this case scenario ?

Re: What types of data ingestion I can use ?

Shelton — Thu, 13 Dec 2018 08:01:03 GMT

In the above Architecture, there is no nifi !! Apache NiFi and Apache Kafka are two different tools with different use-cases that may slightly overlap.

Combining tools may appear wasteful, as it seems to introduce some overlap in functionality. For example, both NiFi and Kafka provide brokers to connect producers and consumers. However, they do so differently: in NiFi, the bulk of the data flow logic lays not inside the producer/consumer, but lives in the broker, allowing for centralized control.

NiFi was built to do one important thing well: data flow management. With both tools combined, NiFi can take advantage of Kafka’s reliable stream data storage, while taking care of the data flow challenges that Kafka was not designed to solve.

HTH

If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer.

Re: What types of data ingestion I can use ?

Shelton — Sat, 17 Aug 2019 22:48:39 GMT

@max mouse

There isn’t a one-and-only tool that can do everything equally well and address all of your requirements. Combining tools that do different things in better ways allows for a buildup in functionality and increased flexibility in handling a larger set of scenarios. Depending on your needs, both NiFi and Flume can act as Kafka producers and/or consumers.

HTH