SQL loader using PutSQL

[#1]

Hi all,

Am trying to build SQL loader using NiFi, the thing which is a bit strange is that I have to split the input file (containing insert statments) to multiple files each containing only one SQL insert, is this the best practice ? why I cant send the whole file to putSQL processor ?

Thanks

[#2 · Accepted solution]

PutSQL has a mechanism for batching together statements that were split by processors such as SplitText. Set the "Support Fragmented Transactions" property to true, and PutSQL will wait until all flow files with the same fragment.identifier have arrived, then it will process them all as a single batch.

There has also been talk of implementing the same improvement for PutSQL as is being done for PutHiveQL (NIFI-3031), to support multiple statements from a single flow file. Please feel free to file a Jira for this if you like.

[#3]

Thanks Matt, very helpful

is there anyway to use "Load into" instead of SQL insert ?

[#4]

Yes, PutSQL accepts any SQL statement (except Callable statements like stored procedures) that does not return a result set, so DDL/DML commands such as LOAD INTO or CREATE TABLE, etc. are supported.

[#5]

big thanks really...