- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to keep track of flowfiles going through a failure relation through nifi data provenance.
- Labels:
-
Apache NiFi
-
NiFi Registry
Created 03-16-2023 06:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I have a requirement where I want to check any flow files going through failure relationships through data provenance directly.
Created 03-16-2023 09:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@ramMuka Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @cotopaul @SAMSAL and @MattWho who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 03-17-2023 06:03 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Assuming that there would be a way to check for flow files which went through failure, how would you extract them? Manual search within the Data Provenance Menu from within the GUI? Or using the REST API?
Now, what I do know is that you cannot identify those files that easy, because they do not write any specific lines within the Data Provenance. If you have a processor right after the failure queue (or if you terminate the failure queue in the processing processor) you can query using that Component ID and identify the type = DROP, meaning that those files have been "processed" (Indicates a provenance event for the conclusion of an object's life for some reason other than object expiration).
More about the types can be found here: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.2/bk_user-guide/content/provenance_events.html
