@fnimi  Yes a write lock is still created  when you use version control. So you'll still deal with read request stacking up until all jetty threads are full resulting in the 503.  The large commit has to be completed successfully before status within NiFi Ui can reflect the current version control status on the Process group.  Version controlled process groups also remove parses the json to remove all sensitive properties.  Version controlled dataflows can also contain parameter contexts if used and controller services they utilize making them even larger. Having 1000+ components in a single version-controlled Process Group is considered an anti-pattern in NiFi. It makes version control, deployments, and UI responsiveness incredibly slow.  Version controlled flow are meant for easy reuse or re-deployment to other NiFi clusters.  Version controlling such large flows reduce the reusability of them.  Modularize your Flows (Nested Versioning) Instead of versioning the top-level Process Group that contains everything, break your flow into smaller, logical, nested Process Groups (e.g., 50–100 components each).  Avoid versioning a PG  within a PG that is already version controlled. You can version control these smaller Process Groups individually. This drastically reduces the serialization time, lock duration, and payload size sent to the Registry. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nissar  Do you observe same issue if you run it more frequently so it is constantly checking for new files to list rather then larger batches every 30 minutes? Matt   ... View more

@fnimi  When you say "make a new version of", are you referring to committing a new version of a version controlled process group to the NiFi-Registry service or creating a new NiFi template?  NiFi templates consume a lot of heap memory which is the leading reason why they were deprecated in Apache NiFi 1.x and fully removed in Apache NiFI 2.x major releases. Do each of these copied 4-5 big process groups contain 300 - 1000 processors or 300 - 1000 total.   When you copy components on the canvas you are creating a flow snippet that contains all the components, configurations, connections, etc of what you have selected.  That snippet is held in heap memory and when you paste it, it creates a request that contains that large snippet that must be replicated to all the nodes in a NiFi cluster.  Also when you paste NiFi must iterate through all the components in that snippet to calculate and set new x,y coordinates, component UUIDs, remap connections because of UUID changes and instantiate those components on the canvas. Also keep in mind that even after all components are added from the paste operation they are then passed along to the validator to check if each has a valid configuration and referenced services are good state and enabled. This operation creates an exclusive write lock until the above completes.  Meanwhile, your browser (and the browsers of any other users that may be open to this NiFi) continues to fire status update requests every so many seconds. These read requests are blocked waiting for the Write Lock to release.  The Jetty service has a thread pool where these read request start stacking up until no additional can be accepted and thus 503 can be encountered.  Also things like CPU thread concurrency, disk I/O, etc utilized during this large request ma be under contention. Instantiating flows from NiFi-Registry or via flow definitions imports is going to use more efficient methods then an in browser copy and paste.  I would avoid copy and paste of large flows and focus on copy paste of smaller snippets at a time.   Also want to note that Apache NiFi 1.19 is more then 3.5 years old and to keep up with critical CVEs and bug fixes you just plan regular upgrades to newer versions. The Apache NiFi 1.x major release branch is end of life and no longer receiving any new updates, fixes, or  security CVE changes in Apache.  The new Apache NIFi 2 major release branch is what is being supported in Apache now. Hope this helps explain your observations: Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  Why are you posting the same response over and over again every few hours? It makes the thread unnecessarily noisy. Your issue is a timeout issue within the SMBJClientProviderService that is attempting to establish a connection with your target SMB share.  This is taking longer then the configured timeout.  The subsequent request is probably working because the connection is still established at time of second run.  Same issue with the FetchSMB it appears as the connection is closed at time of attempting to fetch the File content.  Have you tried increasing the Timeout setting in that controllers service from the default 5 secs to 20 seconds? Have you inspected your network latency between NiFi host and SMB server?   Are you seeing any packet loss? Have you inspected the SMB server logs in Windows? Does the behavior change if you stop using the "run once" option and just start the processor?  Your run schedule of every 30 seconds is allowing the connection to timeout.  If you also adjust that run schedule to 10 seconds on ListSMB, do you see a change in behavior? Unfortunately I do not have a SMB share available to test this setup myself and am providing what guidance and suggestions I can to help with your community query. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@oka  I believe your issue is that "-" is not a valid character in the JMS specification. https://stackoverflow.com/questions/30022453/how-to-set-content-type-header-for-jms-message While the "-" is valid for AMQP messages. There is a ConsumeAMQP processor but I see it only supports AMQP version 0.91 You don't have a issue connecting but just sound like these specific headers are ignored since they contain that invalid JMS character. There is an existing Apache NiFi jira (https://issues.apache.org/jira/browse/NIFI-14670) for adding AMQP 1.0 support to the AMQP processors; however, it is open and unassigned. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fnimi   I am not clear on "It dies for 15-30 minutes".    What version of Apache NiFi are you using? Is this a standalone instance of NiFi or a multi-node NiFi Cluster? Does the NiFi child process die and get launched agin by the NiFi bootstrap process?  Any exceptions thrown in the nifi-app.log or nifi-bootstrap.log (timeouts, OutOfMemory(OOM), etc)?  Are you seeing a long stop-the-world Java Garbage Collection event? How large are these "big" process groups? The more detail you can provide, the better chance a community member will be able to provide some suggestions. Thank you, Matt ... View more

@nisaar  I see you set retry on the ListSMB processor "success" relationship.  You should never set retry on any "Success" relationship.  The connection retry logic is triggered when a FlowFile would be routed to the relationship.  Instead of routing the FlowFile to the relationship it remains on the upstream connection to be "retried".  That means that only when FetchSMB successfully lists a file will it trigger retry.  Plus retry implies a there is some upstream location to persist a source FlowFile. Only once all retry attempts are exhausted will the FlowFile actually be placed on the connection containing the "success" relationship. I would expect unexpected behavior with such a configuration.  Using "retry" on a failure or other error type relationship makes sense since these are the relationships you don't expect FlowFiles to be routed to.  so makes sense on your FetchSMB processor so that source FlowFile will remain on the inbound connection to FetchSMB until both the configured retry attempts are made.  I both retries fail I see you terminate the "Failure" relationship which effectively throws away that file.   I don't know that terminate is what i would do here either since ListSMB was successful in order to produce FlowFile waiting at FetchSMB.  Plus state is recorded by ListSMB and the file that failed ot be fetched would never get listed again unless you touched file to update last modified time or cleared state on listSMB processor.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  What version of Apache NIFi is being used? How is your PutSFPT processor configured? Can you share the complete stack trace from the nifi-app.log? Matt ... View more

 @nisaar  Just so I am clear in your dataflow setup... - ListSMB  processor configured to use cron scheduling (get scheduled for ru every ~30 mins) - ListSMB "success relationship" routed via a connection to FetchSMB processor - FetchSMB configured to use Timer Driven scheduling with:   - Flow looks like this:   The "retry" when set on a relationship controls whether a FlowFile remains on the inbound connection to the processor or gets routed immediately to the destination relationship. The number of retries is how many attempts will be made to reprocess the source FlowFile before finally routing to the destination relationship.  You have 2 so that FlowFile will get terminated if it is not successfully fetched after 2 failed attempts. I am not clear on this statement you made: On first run it fails to either list/fetch the file and the retry kicks in and the files is listed and fetched successfully. There is no "retry" on the listSMB.  It simply get scheduled to run at the configured run schedule and listed based upon previous stored state and processor property configuration. 1. How are files being added to the source SMB directory? 2. As files are added how is the the last modified timestamp being updated?  (if they are being moved to the SMB as an atomic move, the timestamp on the file may not change which can result in ignored files because another file already listed resulted in sate holding a more recent timestamp). 3. What "Listing strategy" are you using?  I recommend using "Tracking Entities" to avoid issue from question 2 above. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@nisaar  When you execute the SFTP CLI, are you doing so as the same user that owns the running NiFi process?  If not, switch to teh NiFi service user and then try your SFTP CLI command again to see if it is successful or if it ask you to accept the host key for the SFTP target.  If you are asked to accept the host key, try running your NiFi putSFTP processor again after you accept the host key on the NiFi service user account. It does not matter what user is authenticated in NiFi or which user created the dataflow on the NiFi canvas.  All components are executed as the NiFi service user.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more