Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can NiFi be used to pipe the data from Oracle Database to Kafka ?

avatar

I want to ingest data from Oracle database to Kafka using Logstash. Wondering if NiFi can perform the ingestion from relational database to Kafka ?

1 ACCEPTED SOLUTION

avatar

Hi, let me make sure I understand the environment. According to https://www.elastic.co/guide/en/logstash/2.0/plugins-inputs-jdbc.html there's nothing in the jdbc plugin to track incremental runs (e.g. only pick up data added/changed since the last run). This has to be built into the query itself (a generic mechanism).

For this matter NiFi can absolutely serve the purpose, not sure there would be much that LogStash would bring on top.

There is also another use case, when Oracle transaction log is being followed for real-time replication, which is a very different architecture. E.g. Oracle GoldenGate is one of the best products in its class (I think it works with other DBs as well).

View solution in original post

7 REPLIES 7

avatar

Hi, let me make sure I understand the environment. According to https://www.elastic.co/guide/en/logstash/2.0/plugins-inputs-jdbc.html there's nothing in the jdbc plugin to track incremental runs (e.g. only pick up data added/changed since the last run). This has to be built into the query itself (a generic mechanism).

For this matter NiFi can absolutely serve the purpose, not sure there would be much that LogStash would bring on top.

There is also another use case, when Oracle transaction log is being followed for real-time replication, which is a very different architecture. E.g. Oracle GoldenGate is one of the best products in its class (I think it works with other DBs as well).

avatar

Thanks @agrande@hortonworks.com. Yes. the source data is just a logs of event. Customer do not want to use any paid software(eg GG )

avatar

avatar

@orenault@hortonworks.com. yes but doesnt appear to provide any opensource options.

avatar

Sqoop2 can do the move to kafka.But we are not there yet to support it .

avatar
Rising Star

@jramakrishnan Do you plan to support Sqoop2 in the near future?

avatar
New Contributor

Keep in mind also there is no upgrade path from Sqoop (HDP included) to Sqoop2