@Dave0x1  Add Encrypt and Decrypt PGP Processors and Services The EncryptContentGPG processor functions in accordance with the openGPG specification defined in RFC 4880.  The specification has changed since then refreshed specification for OpenPGP , which have resulted in changes in the latest Apache NiFi 2.0 milestone releases:  Remove Compression from EncryptContentPGP and SignContentPGP Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Althotta  NiFi node disconnections are rarely the result of some underlying issue in NiFi code.  Node disconnection are more commonly the result of resource consumption or configurations not being optimal in a NiFi deployment.   This particular post if two years old.  You should create a new post and provide details around your node disconnection issue and the specific Apache NiFi version you are running to get better assistance.  The NiFi cluster UI is a good place to start.  The "view details" icon to the far left for each node will provide you with node events which will include disconnect events along with reason.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@kk-nifi  Replay is not the proper way to handle failures.  Failures should be handled in real-time through dataflow design.  The "replay" option is only possible if NiFI still holds the content of the FlowFile you want to replay in its content repository.   The replay ability is really built with the intention to be used in dataflow development testing ( Replaying a FlowFile ) Replay also required numerous manual steps making it difficult to automate retry. - First you need to execute a Provenance query. - From list of provenance events select the event(s) you want to replay one by one. - If content is still available you will have option to "replay" that FlowFile. There is also another option to "Replay last event", but again only works if last FlowFile's content still exists in the NiFi node's content repository.  In your case, you talk about multiple failed FlowFiles for which this will not work to replay them all. NiFi stores the content in content claims with the NiFi content_repository.  A content claim can hold the content for 1 too many FlowFiles.  Once all FlowFiles referencing a content claim have reached point of auto-terminate, the claimant count would be zero.  At that point the content claim will either be moved to archive or deleted depending on archive configuration.   Even if archived, it is only retained for a limited amount of time. Also keep in mind that replay is NOT taking the original FlowFile and replaying it.  Replay generates a new FlowFile with all the same FlowFile attributes and same content as the original FlowFile. Dataflow programatic handling is better.  On failure configure auto retry as i described or route failure to some other processor(s) (optionally for tagging, updating, etc ) and then route back to failed processor.  or better yet configured  X number of auto-retry that only routes to "failure" relationships if all retry events end up failing. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Tiger_Name  The issue you described matches up very well with https://issues.apache.org/jira/browse/NIFI-10018 which caused by https://issues.apache.org/jira/browse/NIFI-9988. While these jiras point at an issue in Apache NiFi 1.16.1, you are using the first milestone (M1) releases of Apache NiFi 2.0.0.  So it quite possible that this issue may have been ported into the early milestone release.   I would recommend downloading the latest milestone release of Apache NiFi 2.0.0-Mx, which at time of writing this is M4, to see if startup issue still presents itself. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@rich197  Providing the exact steps you perform to reproduce might help here.  I am not clear in this step: "Next I attempted to upload a flow into a process group".   Are you trying to upload a template  or a flow definition?   A flow definition is a Process group containing one or more components.  A template can be just a collection of components (Templates are deprecated and removed as of Apache NiFi 2.x) Thank you, Matt ... View more

@varungupta  This is a ~3 year old post with an already accepted answer.  You are likely to get more responsive answers if you were to start a new thread.  NiFi would have also evolved considerable over the past 3 years. Yes, tracking entities does not rely on timestamps to ensure listing of new FlowFiles and will help you here.  NiFi grabbing 1 -2  of 20 is more then just timestamps, I suspect that how the files are being moved into the consumption directory is also impacting you. Tracking Timestamps is easiest and least resource consumption default setup, but does not work for all use cases.   Timestamp is based on the last modified timestamp.  When listing is performed it lists all Files with last processor state stored timestamp up to most recent file's last modified timestamp.  Problem can happen if last modified timestamp is not updated.  For example some system writes to directory A on your local machine and after write completes, it moves file to Directory B.  With that atomic move the file timestamp is not updated.  If the move does not happen fast enough it may get missed in the current listing. it is also possible that a moved file has an older last modified timestamp that another smeller files moved quicker to dir B.  Thus resulting a timestamp stored in state that would be newer and thus resulting in that other file being ignored. Tracking entities was added to  solution to these types of problems. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@kk-nifi  Is there more to your use case and dataflow design you can share? Where in your dataflow is the failure happening? With a successful operation a FlowFile is routed in majority of the processors to a "success" relationship.  Any failure results in a FlowFile being routed to a failure  or retry relationship with most processors.   While in older versions of NiFi users would typically build a retry dataflow from the failure and/or retry relationships.  With the latest versions of NiFi "retry" has been systematically built directly in to the Processor.  The existing capability still exists but this give you even more ability in how you want to handle retry. Example: When you select retry on a relationship like "failure" or "retry" (never select retry on success), you are given the option to specify a retry attempts, a back off policy, and max back off period. Rather then a FlowFile being routed to the "failure" relationship when "retry" is selected, the FlowFile remains on the incoming queue to be tried again (in above example, 10 attempts will be made before the FlowFile is finally routed to the "retry" relationship.).  The Retry Back Off Policy controls how you want to handle these retries:  - "Penalize" applies a penalty to the FlowFile on the inbound connection.  NiFi ignores penalized FlowFiles and continues to execute on other non penalized FlowFiles until the penalty expires. - "Yield"  triggers the processor to yield for a duration of time and then retry the current FlowFile.  This method ensure order of processing as no other FlowFile will be processed until this one is either successfully retried or all retry attempts  have been exhausted and FlowFile has finally been routed to the "failure" relationship. The Retry Max Back Off Period controls the maximum time the FlowFile will either be penalized or max time processor will yield between retry attempts.  Penalty and yield initial time is controlled by the "Yield Duration" and "Penalty Duration" configured in the processor's "Settings" tab. The duration is repeatedly doubled with each retry until the max backoff period is reached. -------------- Now when it comes to NiFi in a "distributed approach", I assume you mean when  you have setup a multi-node NiFi cluster? In a multi-node NiFi cluster, each node loads its own copy of the dataflow and processes FlowFiles that are on that same node.  Node 1 is completely unaware of the specific related to any FlowFile that exists on Nodes 2, 3, 4, etc.. So you need to account for this NiFi architecture in your dataflow designs.  If there is a required order of execute for some batch of FlowFiles, you'll want to keep that batch on the same NiFi node and make sure you are configuring proper "Prioritizers" on all the connections between processor components. There is very little detail in your failure handling.  Why are you cloning?  What is the purpose of the added UpdateAttribute processor? Hopefully the newer "retry" options available on all relationships will help you with your use case. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more