Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache NiFi Reconciliation flow

Solved Go to solution

Apache NiFi Reconciliation flow

Explorer

Good Afternoon,


I am creating an apache nifi flow in which I want to flow files from Oracle DB to IBM/DB2, I have completed that part of the flow.


I need help with configuring a way to reconcile failed files, I want to know exactly which files failed and the error which caused it. I'm considering using a PUTHDFS to put the failed files into a HIVE TABLE that way I can capture the flow file error. I would then re-enter the failure files back into the flow via GETHDFS.

So, I'm curious is this possible?

Also is there a mechanism in NiFi to identify the singular file that fails in a batch?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Apache NiFi Reconciliation flow

Super Guru

If you are not obtaining keys from the database, not using fragmented transactions, and not rolling back on failure, then you should see the failed flow files in a batch being routed to the failure relationship. If you must configure the processor differently, then the flow files will be treated as a single transaction. In that case, in order to handle individual failures you'll want to not use batches, meaning set PutSQL's Batch Size property to 1.

View solution in original post

1 REPLY 1
Highlighted

Re: Apache NiFi Reconciliation flow

Super Guru

If you are not obtaining keys from the database, not using fragmented transactions, and not rolling back on failure, then you should see the failed flow files in a batch being routed to the failure relationship. If you must configure the processor differently, then the flow files will be treated as a single transaction. In that case, in order to handle individual failures you'll want to not use batches, meaning set PutSQL's Batch Size property to 1.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here