Support Questions

Find answers, ask questions, and share your expertise

Apache NiFi Reconciliation flow

avatar
Contributor

Good Afternoon,


I am creating an apache nifi flow in which I want to flow files from Oracle DB to IBM/DB2, I have completed that part of the flow.


I need help with configuring a way to reconcile failed files, I want to know exactly which files failed and the error which caused it. I'm considering using a PUTHDFS to put the failed files into a HIVE TABLE that way I can capture the flow file error. I would then re-enter the failure files back into the flow via GETHDFS.

So, I'm curious is this possible?

Also is there a mechanism in NiFi to identify the singular file that fails in a batch?

1 ACCEPTED SOLUTION

avatar
Master Guru

If you are not obtaining keys from the database, not using fragmented transactions, and not rolling back on failure, then you should see the failed flow files in a batch being routed to the failure relationship. If you must configure the processor differently, then the flow files will be treated as a single transaction. In that case, in order to handle individual failures you'll want to not use batches, meaning set PutSQL's Batch Size property to 1.

View solution in original post

1 REPLY 1

avatar
Master Guru

If you are not obtaining keys from the database, not using fragmented transactions, and not rolling back on failure, then you should see the failed flow files in a batch being routed to the failure relationship. If you must configure the processor differently, then the flow files will be treated as a single transaction. In that case, in order to handle individual failures you'll want to not use batches, meaning set PutSQL's Batch Size property to 1.