Nifi is responding very slow

[#1]

Hello,

I am using Nifi with maximum ram of 26 GB and when i run my flows, it uses 33 GB. I have used an API mentioned below, to track the status of all my flows,

http://localhost/nifi-api/flow/status

Here is my response from the API

{"controllerStatus":{"activeThreadCount":3,"queued":"52,687 / 27.18 MB","flowFilesQueued":52687,"bytesQueued":28495782,"runningCount":127,"stoppedCount":9918,"invalidCount":3239,"disabledCount":153,"activeRemotePortCount":0,"inactiveRemotePortCount":0}}

After some time it started responding very slow unless i restart it again.I am facing this problem again and again, and i am not able to run my flows properly.

[#2 · Accepted solution]

@Prakhar Agrawal

In your flow status you are having "stoppedCount":9918,"invalidCount":3239 i.e almost 13k processors are not running now(either stopped or invalid).

Even these processors are not running now but NiFi will validate them every time to check the status of the processors until we manually disable those processors.Disable all the stopped/invalid processors in your NiFi instance.

How to disable 13k processors?

Method 1:-

Changing in flow.xml.gz

1.Stop the NiFi instance(to make sure no new processors are adding at the time of changing flow.xml)

2.take a backup of flow.xml.gz

3.guzip flow.xml.gz

4.Search for <scheduledState>STOPPED</scheduledState> in flow.xml and replace with <scheduledState>DISABLED</scheduledState>

5.gzip flow.xml

6. Then you need to replicate the same flow.xml.gz to all other nodes of your HDF cluster (or) just rename the flow.xml.gz to some other names like bkp_flow.xml.gz_<time>.

7.Start NiFi
(or)

Method2:-

Create a flow in NiFi instance in this way you don't need to stop/start the NiFi instance

1. Read flow.xml.gz

2. Parse XML to find stopped processors and their IDs (as per above)

3. Use NiFi rest API to change a state to disabled.

-

For more details refer this article to Improving performance of NiFi UI

[#3]

Thanks Shu for quick response.

According to you, I can disable all invalid processor but I won't able to disable all stopped processor as 90% stopped processor in my nifi are useful and I can run it on time to time. And there may be possibilities of increase in no. of processor.

When I check memory usage using "system-diagnostics" rest api then it give me below output:

{"systemDiagnostics":{"aggregateSnapshot":{"totalNonHeap":"831.29 MB","totalNonHeapBytes":871669760,"usedNonHeap":"796.05 MB","usedNonHeapBytes":834719376,"freeNonHeap":"35.24 MB","freeNonHeapBytes":36950384,"maxNonHeap":"-1 bytes","maxNonHeapBytes":-1,"totalHeap":"19.55 GB","totalHeapBytes":20992491520,"usedHeap":"8.25 GB","usedHeapBytes":8862073056,"freeHeap":"11.3 GB","freeHeapBytes":12130418464,"maxHeap":"19.55 GB","maxHeapBytes":20992491520,"heapUtilization":"42.0%","availableProcessors":8,"processorLoadAverage":10.91,"totalThreads":428,"daemonThreads":201,"uptime":"00:23:16.322","flowFileRepositoryStorageUsage":{"freeSpace":"314.39 GB","totalSpace":"1.43 TB","usedSpace":"1.12 TB","freeSpaceBytes":337573191680,"totalSpaceBytes":1572096000000,"usedSpaceBytes":1234522808320,"utilization":"79.0%"},"contentRepositoryStorageUsage":[{"identifier":"default","freeSpace":"314.39 GB","totalSpace":"1.43 TB","usedSpace":"1.12 TB","freeSpaceBytes":337573191680,"totalSpaceBytes":1572096000000,"usedSpaceBytes":1234522808320,"utilization":"79.0%"}],"provenanceRepositoryStorageUsage":[{"identifier":"default","freeSpace":"314.39 GB","totalSpace":"1.43 TB","usedSpace":"1.12 TB","freeSpaceBytes":337573191680,"totalSpaceBytes":1572096000000,"usedSpaceBytes":1234522808320,"utilization":"79.0%"}],"garbageCollection":[{"name":"G1 Young Generation","collectionCount":1471,"collectionTime":"00:05:30.811","collectionMillis":330811},{"name":"G1 Old Generation","collectionCount":0,"collectionTime":"00:00:00.000","collectionMillis":0}],"statsLastRefreshed":"20:06:22 IST","versionInfo":{"niFiVersion":"1.4.0","javaVendor":"Oracle Corporation","javaVersion":"1.8.0_77","osName":"Linux","osVersion":"3.10.0-514.el7.x86_64","osArchitecture":"amd64","buildTag":"nifi-1.4.0-RC2","buildTimestamp":"12/04/2017 19:55:41 IST"}}}}

As you can notice that, I have provided max 20GB memory. So, I think this is enough memory to run nifi smoothly with 15k+ no. of processor. As there is no limit of no. of processor dragged on canvas then what can I don to run nifi smoothly with 15k+ or 20k+ processors. Please let me know as soon as possible.

Thanks in advance.

[#4]

@Prakhar Agrawal

Shu's answer deals purely with the responsiveness of the NiFi UI and has nothing to do with the performance of running dataflows. Disabling a processor does not mean it is removed from the canvas. A processor component in a disabled state is not validated by the NIFi controller. You can enable and run processors anytime you like and then stop and disable them when done. At a minimum you should disable all your invalid processors since they are not capable of running anyway.
-

So the real question is what you mean by "responding very slow" specifically the canvas or do you notice flows processing Flowfiles slower. How did you come to that conclusion?

-

This may come down to tuning your dataflow(s) themselves.

What is "Max Timer Driven Thread Count" set to in "controller settings" found in global menu. Default is only 10. Which means all processor must share only 10 cpu threads.
Processors that appear to be processing slowly, how many concurrent tasks have they been configured to use, run schedules, etc...?

-

Thanks,

Matt

[#5]

@Matt

Thanks for reply.

My point behind "Nifi Slow Response" is UI gives slow response in some conditions i.e.

1. When I open configuration model window of any processor.

2. When I click on apply button after configuration.

3. When I am going to open controller services to configure one of my controller.

In all the above cases sometimes it works fine and respond me in a second or two but sometime it take 20 sec to 1 min or more then 1 minute. Now I want to know that I have enough memory still I am facing this type of response.

And I use my NIFI with KYLO so I wont able to disable my processor, Is there any other way to solve this problem?

Max Timer Driven Thread Count is set to 20

Max Cron Driven Thread Count is set to 50.

Thanks in advance

[#6]

@Matt

Thanks for reply.

My point behind "Nifi Slow Response" is UI gives slow response in some conditions i.e.

1. When I open configuration model window of any processor.

2. When I click on apply button after configuration.

3. When I am going to open controller services to configure one of my controller.

4. When I reload my Nifi instance.

In all the above cases sometimes it works fine and respond me in a second or two but sometime it take 20 sec to 1 min or more then 1 minute. Now I want to know that I have enough memory still I am facing this type of response.

And I use my NIFI with KYLO so I wont able to disable my processor, Is there any other way to solve this problem?

Max Timer Driven Thread Count is set to 20

Max Cron Driven Thread Count is set to 50.

Thanks in advance

[#7]

@Prakhar Agrawal

Then your issue is going to be the backend validation that is going on. How much validation that is occurring depends on what you are doing. Shu's recommendation for disabling processors that are not running should help a great deal.

-

There is no such thing as "Max Cron Driven Thread Count". I am going to assume you meant "Max Event Driven Thread count." There is nothing you will add to your canvas that will use the "Event Driven" scheduling strategy. You physically need to change the configuration on a component to use "Event Driven". Event Driven is consider experimental and deprecated in favor of advance made with the Timer driven Scheduling strategy. By setting "Max Event Driven Thread count" to 50 you are creating a pool of 50 reserved threads. I recommend not using "Event Driven" Scheduling strategy anywhere and reducing the max Event Driven thread count setting to "1". Reducing the "Max Event driven thread count" setting will require a NiFi restart.