Created 05-06-2019 05:18 PM
Good Afternoon,
I am creating an apache nifi flow in which I want to flow files from Oracle DB to IBM/DB2, I have completed that part of the flow.
I need help with configuring a way to reconcile failed files, I want to know exactly which files failed and the error which caused it. I'm considering using a PUTHDFS to put the failed files into a HIVE TABLE that way I can capture the flow file error. I would then re-enter the failure files back into the flow via GETHDFS.
So, I'm curious is this possible?
Also is there a mechanism in NiFi to identify the singular file that fails in a batch?
Created 05-07-2019 01:30 PM
If you are not obtaining keys from the database, not using fragmented transactions, and not rolling back on failure, then you should see the failed flow files in a batch being routed to the failure relationship. If you must configure the processor differently, then the flow files will be treated as a single transaction. In that case, in order to handle individual failures you'll want to not use batches, meaning set PutSQL's Batch Size property to 1.
Created 05-07-2019 01:30 PM
If you are not obtaining keys from the database, not using fragmented transactions, and not rolling back on failure, then you should see the failed flow files in a batch being routed to the failure relationship. If you must configure the processor differently, then the flow files will be treated as a single transaction. In that case, in order to handle individual failures you'll want to not use batches, meaning set PutSQL's Batch Size property to 1.