Hi,
System: NiFi 1.19, Java 11, Win 64bit, 610 processors, 245 running
I have a problem at my current running flows.
Example: the LogMessage-Processor shows, that there are 112 flows in the last 5 minutes
But if I select "view data provenance", there are no flows displayed
As you can see, the "Oldest event available: 08/29/2023", this is one month ago.
I checked the hard disk, where NiFi is running: there is 25 GB free memory, so it shouldn't be a full-disk-problem
Any idea, what we have mis-configured?
@Frank37
Welcome to the community.
NiFi's provenance tracks specific types of provenance events and not all processors produce these types of events. So typically a processor like LogMessage would not be producing any provenance events. Provenance is used to track the lifecycle of a NiFi FlowFile tracking any time it is routed, modified (content or attributes), cloned, dropped, etc.
That being said, it appears your LogMessage processor is auto-terminating it's success relationship? In that case it should be generating "DROP" provenance events.
I setup a simply flow to verify this works successfully:
So your issue seems unrelated to the processor specifically.
First thing to check are your provenance related configuration properties in the nifi.properties file. You stated oldest event is more than a month old which is not expected unless you change your default provenance settings which only retains provenance events for 7 days.
If you go to the global menu in the upper right corner of the UI and select "Data Provance", do you get any provenance results? If so, the newest will be at the top. What is the dat on the newest event? If it is old and you have active running flows, this tells me provenance has not been updating for some time. Does newest provenance event align with some change to your NiFi or upgrade/migration?
maybe a configuration or corruption issue:
Which provenance persistence provider are you using?
What is the newest file in your provenance repository storage directory?
I see you are running Apache NiFi 1.19. Was this instance always running this version or did you upgrade at some point in time? If so what version was used previously?
There were some bugs in older versions of NiFi that could have rendered the provenance repository corrupt if this was upgrade from some older version at one point in time. Stopping NiFi, purging all contents of provenance repository and starting up would resolve that.
Maybe an authorization issue:
Has your user been authorized to "view provenance" data on this component or on the process group in which this processor resides? The authorization to "query provenance" is not that same as the authorization to "view provenance" events produced by individual components. Processors inherit authorization from the parent process group unless explicit policies have been set directly on the component. Process Group in turn inherit form parent Process groups if explicit policy is not set on child process group.
If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
@Frank37
Welcome to the community.
NiFi's provenance tracks specific types of provenance events and not all processors produce these types of events. So typically a processor like LogMessage would not be producing any provenance events. Provenance is used to track the lifecycle of a NiFi FlowFile tracking any time it is routed, modified (content or attributes), cloned, dropped, etc.
That being said, it appears your LogMessage processor is auto-terminating it's success relationship? In that case it should be generating "DROP" provenance events.
I setup a simply flow to verify this works successfully:
So your issue seems unrelated to the processor specifically.
First thing to check are your provenance related configuration properties in the nifi.properties file. You stated oldest event is more than a month old which is not expected unless you change your default provenance settings which only retains provenance events for 7 days.
If you go to the global menu in the upper right corner of the UI and select "Data Provance", do you get any provenance results? If so, the newest will be at the top. What is the dat on the newest event? If it is old and you have active running flows, this tells me provenance has not been updating for some time. Does newest provenance event align with some change to your NiFi or upgrade/migration?
maybe a configuration or corruption issue:
Which provenance persistence provider are you using?
What is the newest file in your provenance repository storage directory?
I see you are running Apache NiFi 1.19. Was this instance always running this version or did you upgrade at some point in time? If so what version was used previously?
There were some bugs in older versions of NiFi that could have rendered the provenance repository corrupt if this was upgrade from some older version at one point in time. Stopping NiFi, purging all contents of provenance repository and starting up would resolve that.
Maybe an authorization issue:
Has your user been authorized to "view provenance" data on this component or on the process group in which this processor resides? The authorization to "query provenance" is not that same as the authorization to "view provenance" events produced by individual components. Processors inherit authorization from the parent process group unless explicit policies have been set directly on the component. Process Group in turn inherit form parent Process groups if explicit policy is not set on child process group.
If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
