question Execute sqoop on NiFi in Archives of Support Questions (Read Only)

Execute sqoop on NiFi

regie_canada — Tue, 24 Jan 2017 14:20:03 GMT

Is there any NiFi processor i can use in executing SQOOP?

If none, any data flow i can use in getting my table then save it to HDFS?

Thanks.

Re: Execute sqoop on NiFi

pvillard — Tue, 24 Jan 2017 17:30:41 GMT

If you really want to use Sqoop, then you would need to use something like ExecuteStreamCommand / ExecuteProcess processors. However, this is not something I'd recommend unless you need the features provided by Sqoop.

If you want a solution fully provided by NiFi, then depending on your source database, you can use the JDBC processors to get the data of your table and then use something like PutHDFS to send the data into HDFS. A common approach is something like GenerateTableFetch on the primary node and QueryDatabaseTable on all nodes. The first processor will generate SQL queries to fetch the data by "page" of specified size, and the second will actually get the data. This way, all nodes of your NiFi cluster can be used to get the data from the database.

You can have a look to the documentation here:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GenerateTableFetch/index.html

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html

You have additional SQL/JDBC processors based on your needs.

This article should get you started:

https://community.hortonworks.com/articles/51902/incremental-fetch-in-nifi-with-querydatabasetable.html

Hope this helps.

Re: Execute sqoop on NiFi

BorisTyukin — Wed, 28 Feb 2018 04:58:25 GMT

Hi @regie canada, check my blog post on this subject

How to run Sqoop from NiFi

Boris

Re: Execute sqoop on NiFi

henrikolsen — Mon, 09 Jul 2018 17:25:57 GMT

@Pierre Villard: "A common approach is something like GenerateTableFetch on the primary node and QueryDatabaseTable on all nodes. The first processor will generate SQL queries to fetch the data by "page" of specified size, and the second will actually get the data. This way, all nodes of your NiFi cluster can be used to get the data from the database.":

Will I need to make a (local) RPG after the GenerateTableFetch to get them running in parallel?

Any experience on performance for making full RDBMS table dumps using this method vs Sqoop?