Can NiFi be used to pipe the data from Oracle Database to Kafka ?

[#1]

I want to ingest data from Oracle database to Kafka using Logstash. Wondering if NiFi can perform the ingestion from relational database to Kafka ?

[#2 · Accepted solution]

Hi, let me make sure I understand the environment. According to https://www.elastic.co/guide/en/logstash/2.0/plugins-inputs-jdbc.html there's nothing in the jdbc plugin to track incremental runs (e.g. only pick up data added/changed since the last run). This has to be built into the query itself (a generic mechanism).

For this matter NiFi can absolutely serve the purpose, not sure there would be much that LogStash would bring on top.

There is also another use case, when Oracle transaction log is being followed for real-time replication, which is a very different architecture. E.g. Oracle GoldenGate is one of the best products in its class (I think it works with other DBs as well).

[#3]

Thanks @agrande@hortonworks.com. Yes. the source data is just a logs of event. Customer do not want to use any paid software(eg GG )

[#4]

Have you looked at : http://www.slideshare.net/gwenshap/kafka-for-dbas

Olivier

[#5]

@orenault@hortonworks.com. yes but doesnt appear to provide any opensource options.

[#6]

Sqoop2 can do the move to kafka.But we are not there yet to support it .

[#7]

@jramakrishnan Do you plan to support Sqoop2 in the near future?

[#8]

Keep in mind also there is no upgrade path from Sqoop (HDP included) to Sqoop2