Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

SQL loader using PutSQL

Hi all,

Am trying to build SQL loader using NiFi, the thing which is a bit strange is that I have to split the input file (containing insert statments) to multiple files each containing only one SQL insert, is this the best practice ? why I cant send the whole file to putSQL processor ?

Thanks

9648-screen-shot-2016-11-20-at-90149-pm.png

1 ACCEPTED SOLUTION

Super Guru

PutSQL has a mechanism for batching together statements that were split by processors such as SplitText. Set the "Support Fragmented Transactions" property to true, and PutSQL will wait until all flow files with the same fragment.identifier have arrived, then it will process them all as a single batch.

There has also been talk of implementing the same improvement for PutSQL as is being done for PutHiveQL (NIFI-3031), to support multiple statements from a single flow file. Please feel free to file a Jira for this if you like.

View solution in original post

4 REPLIES 4

Super Guru

PutSQL has a mechanism for batching together statements that were split by processors such as SplitText. Set the "Support Fragmented Transactions" property to true, and PutSQL will wait until all flow files with the same fragment.identifier have arrived, then it will process them all as a single batch.

There has also been talk of implementing the same improvement for PutSQL as is being done for PutHiveQL (NIFI-3031), to support multiple statements from a single flow file. Please feel free to file a Jira for this if you like.

Thanks Matt, very helpful

is there anyway to use "Load into" instead of SQL insert ?

Super Guru

Yes, PutSQL accepts any SQL statement (except Callable statements like stored procedures) that does not return a result set, so DDL/DML commands such as LOAD INTO or CREATE TABLE, etc. are supported.

big thanks really...