Created 03-16-2023 06:19 AM
Hi,
I have a requirement where I want to check any flow files going through failure relationships through data provenance directly.
Created 03-16-2023 09:57 AM
@ramMuka Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @cotopaul @SAMSAL and @MattWho who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Created 03-17-2023 06:03 AM
Assuming that there would be a way to check for flow files which went through failure, how would you extract them? Manual search within the Data Provenance Menu from within the GUI? Or using the REST API?
Now, what I do know is that you cannot identify those files that easy, because they do not write any specific lines within the Data Provenance. If you have a processor right after the failure queue (or if you terminate the failure queue in the processing processor) you can query using that Component ID and identify the type = DROP, meaning that those files have been "processed" (Indicates a provenance event for the conclusion of an object's life for some reason other than object expiration).
More about the types can be found here: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.2/bk_user-guide/content/provenance_events.html