Member since
02-10-2016
10
Posts
8
Kudos Received
0
Solutions
04-01-2016
12:07 PM
Neither Sqoop nor NiFi are as minimally invasive as for instance Oracle GoldenGate, which checks Oracle's REDO logs for changes rather than fires queries based on a schedule. In many cases, Sqoop or NiFi are fine, but when you need to make sure that the DB is not overloaded by many consecutive requests, it's an idea to look at non-OSS technologies. Some of the open-source alternatives seem to re-invent the elliptic wheel over and over again. Sqoop at least allows you to specify a table (if you want the whole thing), whereas in NiFi you need to specify the query, which is effectively the same just more typing. Sqoop's split-by logic just takes the MIN/MAX of a column, which may not be very efficient (if you have no indexes). I'm not very familiar with the performance of NiFi for relational DBs and the load on those because I haven't found NiFi to be too useful. For transporting the data, it's perhaps fine, but any manipulation is likely going to be very, very tedious. That's what ETL tools have been designed for. Most of them nowadays have connectors for HDFS and the likes. Disclaimer: I do not work for, I am not affiliated with, and I am definitely not paid by Oracle.
... View more