The individual components are responsible for handling any kind fo timeout related to execution of the components code. The NiFi Connections hold NiFi FlowFiles that the downstream processor component will consume from. Typically the FlowFile being executed upon by the downstream component is not removed from the inbound connection until the execution by that downstream connection is complete and its execution has resulted in the creation of its outbound FlowFile (which may be same as ingest FlowFile depending on what that processor does. This is done to protect against dataloss in the even the NiFi service crashes or shut down while a component is still executing. On NiFi service starting back up, the FlowFile would get loaded back in to the same upstream connection allowing the downstream processor to start over executing on the same FlowFile(s).
From you description, it sounds like these custom (not part of Apache NiFi or Cloudera distributions) components are becoming hun while executing against a FlowFile from the upstream connection?
When your flow is in this hung state, does the MarkLogic processor show a small number in the upper right corner of the processor indicating it has an active thread?
Or are you saying the the MarkLogic processor is not actively executing a thread and is not getting a thread to work on the next queued FlowFile? Did you execute "nifi.sh dump <dump-filename>" to verify there is no MarkLogic processor class threads executing?
Is it possible that your entire "Max Timer Driven Thread pool" is consumed by other components on your canvas at times preventing this processor from being able to get a thread to run?
If this turns out to not be a thread starvation issue, you may need to reach out to the author of these custom components for suggestions.
If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.
Thank you,
Matt
The individual components are responsible for handling any kind fo timeout related to execution of the components code. The NiFi Connections hold NiFi FlowFiles that the downstream processor component will consume from. Typically the FlowFile being executed upon by the downstream component is not removed from the inbound connection until the execution by that downstream connection is complete and its execution has resulted in the creation of its outbound FlowFile (which may be same as ingest FlowFile depending on what that processor does. This is done to protect against dataloss in the even the NiFi service crashes or shut down while a component is still executing. On NiFi service starting back up, the FlowFile would get loaded back in to the same upstream connection allowing the downstream processor to start over executing on the same FlowFile(s).
From you description, it sounds like these custom (not part of Apache NiFi or Cloudera distributions) components are becoming hun while executing against a FlowFile from the upstream connection?
When your flow is in this hung state, does the MarkLogic processor show a small number in the upper right corner of the processor indicating it has an active thread?
Or are you saying the the MarkLogic processor is not actively executing a thread and is not getting a thread to work on the next queued FlowFile? Did you execute "nifi.sh dump <dump-filename>" to verify there is no MarkLogic processor class threads executing?
Is it possible that your entire "Max Timer Driven Thread pool" is consumed by other components on your canvas at times preventing this processor from being able to get a thread to run?
If this turns out to not be a thread starvation issue, you may need to reach out to the author of these custom components for suggestions.
If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.
Thank you,
Matt
