Nifi for CDC (With and without PK) from Oracle

[#1]

We have a use case to stream data from Oracle tho HDFS/Hive (Ideally in Real Time) from Oracle DB

1.The Oracle DB Cannot be modified as it is a proprietary Product. We will have access and read privileges to it

2.Data needs to be landed in HDFS and also loaded into Hive

3.Data may be made avail in Kafka as well

Is niFI a good solution to accomplish this.

When we looked at NifI a few releases ago, there were some limitations.

Any one has experience doing this , please chime in.

[#2]

Apache NiFi can read from Oracle, land in HDFS as Apache ORC files for Hive, also send to Kafka.

You can stream data but for Oracle right now there is no connection to its Write Ahead Logs, you have to point to tables and know how they are changing either a datetime increase or sequence id increase. Some people will create triggers to send data to special staging tables.

Or add Attunity to Apache NiFi for pure CDC.

see:

https://community.hortonworks.com/articles/113941/change-data-capture-cdc-with-apache-nifi-version-1...

https://community.hortonworks.com/articles/81153/basic-cdc-using-apache-nifi-and-sap-hana.html

https://community.hortonworks.com/articles/55422/change-data-capture-using-nifi.html

https://community.hortonworks.com/articles/48843/slowly-changing-dimensions-on-hadoop-part-1.html

https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.h...

https://community.hortonworks.com/articles/87632/ingesting-sql-server-tables-into-hive-via-apache-n....

https://community.hortonworks.com/articles/108718/ingesting-rdbms-data-as-new-tables-arrive-automagi...