503 error while copying/versioning big process groups

[#1]

Hello, everytime I make a new version of or copy 4-5 big process groups, the Nifi UI start throwing 503 errors and sometimes It dies for 15-30 minutes. I tried changing the Garbage Collector algorythm but it did not work out, and I know that adding up memory heap will be useless. What else can I do? Thank you

[#2]

@fnimi 

 I am not clear on "It dies for 15-30 minutes".   

1. What version of Apache NiFi are you using?
2. Is this a standalone instance of NiFi or a multi-node NiFi Cluster?
3. Does the NiFi child process die and get launched agin by the NiFi bootstrap process? 
4. Any exceptions thrown in the nifi-app.log or nifi-bootstrap.log (timeouts, OutOfMemory(OOM), etc)? 
5. Are you seeing a long stop-the-world Java Garbage Collection event?
6. How large are these "big" process groups?

The more detail you can provide, the better chance a community member will be able to provide some suggestions.

Thank you,
Matt

[#3]

@MattWho 

Thank you for your reply. What I meant is that the UI gets freezed for that period of time.

1 and 2. I had this issue in a Nifi 1.19 cluster and a Nifi 1.19 standalone.

3. The nifi-boostrap.log does not get updates when this happens, so i think it does not.

4. Reading the standalone's nifi-app.log, I found some SocketTimeoutExceptions at the time of the issue. I did not find out of memory exceptions.

5. No, stop-the-world times are no longer than 100-200 milliseconds.

6. They have from 300 to 1000 processors.

[#4]

@fnimi 

1. When you say "make a new version of", are you referring to committing a new version of a version controlled process group to the NiFi-Registry service or creating a new NiFi template?  NiFi templates consume a lot of heap memory which is the leading reason why they were deprecated in Apache NiFi 1.x and fully removed in Apache NiFI 2.x major releases.
2. Do each of these copied 4-5 big process groups contain 300 - 1000 processors or 300 - 1000 total.   When you copy components on the canvas you are creating a flow snippet that contains all the components, configurations, connections, etc of what you have selected.  That snippet is held in heap memory and when you paste it, it creates a request that contains that large snippet that must be replicated to all the nodes in a NiFi cluster.  Also when you paste NiFi must iterate through all the components in that snippet to calculate and set new x,y coordinates, component UUIDs, remap connections because of UUID changes and instantiate those components on the canvas. Also keep in mind that even after all components are added from the paste operation they are then passed along to the validator to check if each has a valid configuration and referenced services are good state and enabled. This operation creates an exclusive write lock until the above completes.  Meanwhile, your browser (and the browsers of any other users that may be open to this NiFi) continues to fire status update requests every so many seconds. These read requests are blocked waiting for the Write Lock to release.  The Jetty service has a thread pool where these read request start stacking up until no additional can be accepted and thus 503 can be encountered.  Also things like CPU thread concurrency, disk I/O, etc utilized during this large request ma be under contention.
   

Instantiating flows from NiFi-Registry or via flow definitions imports is going to use more efficient methods then an in browser copy and paste.  I would avoid copy and paste of large flows and focus on copy paste of smaller snippets at a time.  

Also want to note that Apache NiFi 1.19 is more then 3.5 years old and to keep up with critical CVEs and bug fixes you just plan regular upgrades to newer versions. The Apache NiFi 1.x major release branch is end of life and no longer receiving any new updates, fixes, or  security CVE changes in Apache.  The new Apache NIFi 2 major release branch is what is being supported in Apache now.

Hope this helps explain your observations:

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

@MattWho I am referring to commiting a new version of a version controlled process group to NiFi-Registry, and each process group contains 300-1000 processors. Does this exclusive write lock also work when versioning using Nifi Registry?

We are planning to move to Nifi 1.28 before upgrading to Nifi 2. Do you think it will be possible to keep versioning process groups of this size without having this issue?

Thank you

[#6]

@fnimi 

Yes a write lock is still created  when you use version control. So you'll still deal with read request stacking up until all jetty threads are full resulting in the 503.  The large commit has to be completed successfully before status within NiFi Ui can reflect the current version control status on the Process group.  Version controlled process groups also remove parses the json to remove all sensitive properties.  Version controlled dataflows can also contain parameter contexts if used and controller services they utilize making them even larger.

Having 1000+ components in a single version-controlled Process Group is considered an anti-pattern in NiFi. It makes version control, deployments, and UI responsiveness incredibly slow.  Version controlled flow are meant for easy reuse or re-deployment to other NiFi clusters.  Version controlling such large flows reduce the reusability of them. 

Modularize your Flows (Nested Versioning) Instead of versioning the top-level Process Group that contains everything, break your flow into smaller, logical, nested Process Groups (e.g., 50–100 components each).  Avoid versioning a PG  within a PG that is already version controlled.

• You can version control these smaller Process Groups individually.
• This drastically reduces the serialization time, lock duration, and payload size sent to the Registry.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt