Many process groups have a success and failure output relationship. A common question is how to best handle these failures. For invalid data, it makes sense to output the flow files to an HDFS directory for analysis, but not when failure was caused by an external dependency (e.g. HDFS, Kafka, FTP).
A simple solution might be to loop the failures back to retry, but then it may fail repeatedly without notifying an administrator. A better solution would be to retry three times, then, if it still has not succeeded, an administrator should be notified and the flow file should wait before trying again. This gives the administrator time to resolve the issue and the ability to quickly and easily retry the flow files.
Below (and attached) is a simple process group that implements this logic. The failed flow files come in through the input port. The UpdateAttribute processor sets the retryCount attribute to one or increments it if it has already been set. The RouteOnAttribute processor determines whether the retryCount attribute is over a threshold (e.g. three). If it is not over the threshold, the flow file is routed out through the retry port. If it is over the threshold, the flow file is routed to a PutEmail processor. The last UpdateAttribute processor should be disabled at all times so that the flowfiles will queue up after the PutEmail processor to wait for the administrator to resolve the issue. Once the issue is resolved, the administrator simply enables, starts, stops, and disables this last processor. The retryCount attribute will be set to zero and the flow file will go out through the retry port. If the flow file still does not succeed, it will go back into this process group and the administrator will get another email.
Note that a merge content processor could be used to reduce the number of emails, if necessary.
