Support Questions

Find answers, ask questions, and share your expertise

SQL loader using PutSQL

avatar
Rising Star

Hi all,

Am trying to build SQL loader using NiFi, the thing which is a bit strange is that I have to split the input file (containing insert statments) to multiple files each containing only one SQL insert, is this the best practice ? why I cant send the whole file to putSQL processor ?

Thanks

9648-screen-shot-2016-11-20-at-90149-pm.png

1 ACCEPTED SOLUTION

avatar
Master Guru

PutSQL has a mechanism for batching together statements that were split by processors such as SplitText. Set the "Support Fragmented Transactions" property to true, and PutSQL will wait until all flow files with the same fragment.identifier have arrived, then it will process them all as a single batch.

There has also been talk of implementing the same improvement for PutSQL as is being done for PutHiveQL (NIFI-3031), to support multiple statements from a single flow file. Please feel free to file a Jira for this if you like.

View solution in original post

4 REPLIES 4

avatar
Master Guru

PutSQL has a mechanism for batching together statements that were split by processors such as SplitText. Set the "Support Fragmented Transactions" property to true, and PutSQL will wait until all flow files with the same fragment.identifier have arrived, then it will process them all as a single batch.

There has also been talk of implementing the same improvement for PutSQL as is being done for PutHiveQL (NIFI-3031), to support multiple statements from a single flow file. Please feel free to file a Jira for this if you like.

avatar
Rising Star

Thanks Matt, very helpful

is there anyway to use "Load into" instead of SQL insert ?

avatar
Master Guru

Yes, PutSQL accepts any SQL statement (except Callable statements like stored procedures) that does not return a result set, so DDL/DML commands such as LOAD INTO or CREATE TABLE, etc. are supported.

avatar
Rising Star

big thanks really...