question Re: Hortonworks Data Flow (Apache Nifi) in Archives of Support Questions (Read Only)

Hortonworks Data Flow (Apache Nifi)

techiegreenhorn — Tue, 21 Apr 2026 12:32:41 GMT

What is the best mechanism to ingest data from relational sources into HDP. To use a combination of ExecuteSQLand putHDFS processors or to use Sqoop and deliver the data to HDP?

Thanks

Re: Hortonworks Data Flow (Apache Nifi)

nsabharwal — Fri, 12 Feb 2016 16:47:46 GMT

@Greenhorn Techie

You can definitely use Sqoop and it's part of HDP stack.

You can leverage HDF (NiFi) to ingest data into HDP. I this case you have to get on support for HDF and HDP.

Re: Hortonworks Data Flow (Apache Nifi)

techiegreenhorn — Fri, 12 Feb 2016 18:49:12 GMT

@Neeraj Sabharwal Thanks. My question, which tool is best placed to handle data loading from RDBMS. I understand both of them support. But I would like to understand which one is more capable and advantageous over the other.

Thanks

Vijay

Re: Hortonworks Data Flow (Apache Nifi)

nsabharwal — Fri, 12 Feb 2016 18:50:23 GMT

@Greenhorn Techie Sqoop is the most useable tool in the industry as of today for this use case.

Re: Hortonworks Data Flow (Apache Nifi)

techiegreenhorn — Fri, 12 Feb 2016 20:10:14 GMT

Thanks @Neeraj Sabharwal for validating my understanding 🙂

Re: Hortonworks Data Flow (Apache Nifi)

vvaks — Mon, 28 Mar 2016 06:45:43 GMT

I would use Sqoop for ingesting RDBMS data as Sqoop will parallelize the ETL job while Nifi will simply run it on the thread that the processor is running on. To do the same thing with Nifi, you would have to create multiple instance of the executeSQL processor and go after partitions of the data you are after.

Re: Hortonworks Data Flow (Apache Nifi)

aervits — Mon, 28 Mar 2016 08:22:51 GMT

Nifi 0.6 release adds ability to run simple change capture cases with QueryDatabaseTable by maintaining timestamps. This might start turning the needle towards nifi away from sqoop https://cwiki.apache.org/confluence/display/NIFI/Release+Notes