Hi ,
I need cloudera's\community's help with this because I have been struggling with it for some time. we got new VMs that going to be dedicated to Nifi. We have been using Nifi 2.0.0 M1 on an old physical servers with custom developed python custom extensions and we did not see any issues. When we moved to the new VM servers ( See spec below) at that time M2 was released and we decided to go with the latest, however once its installed and we got it working we started noticing the CPU utilization Peaks to almost 99-100% ! When I check the Processes under task manager I notice this utilization is taken by Java that is running Nifi:
This happens even without running anything on Nifi. I reverted back to M1 release thinking maybe its an M2 issue since M1 did not have this issue on the old physical servers, however the problem persisted even when using the same Java & Python versions.
After some troubleshooting and going through process of elimination, I found that if I dont enable the python extension or dont deploy any of those extensions ( while its enabled) the issue is gone !!! It seems there is something with the python extension that is causing the CPU to peak. One thing I noticed when I check the task manager is too many python processes even though nothing is running at that point, and they dont seem to use much of the CPU however having them in that number seem to be strange :
The more I add python extension those processes kind of double in number! In the old server (M1 release ) I counted over 35 python processes running and I only have about 8 extensions! Nothing was running in Nifi at that time.
Im going to try and test this with M3 release since there have been some changes and bug fixes when it comes to python extension to see if the issue persist.
Has anyone run into this issue before? If so can you please advise. If not, Is there a way at least to get to the bottom of this and know exactly what is happening or what is causing it. Anything to help with providing our IT with something to work with would be highly appreciated.
The server spec Im running Nifi on:
Thanks
Dear @steven-matison ,
Let me say first you are AWESOME for going as far as you did to help me figure this thing out. I cant thank you enough.
I think my lesson learned here is that I should not rely only the release notes to see if a major issue like this has been addressed and probably always review the bug fixes under Jira to help me decide wither an upgrade is worth it or not. However having said that , I still believe something as critical as memory leak and high CPU utilization should have been mentioned as part of the highlights: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3
I really wish I knew this thing earlier, it would have saved me days of troubleshooting and the stress of being under the watch of IT because they get a notification every time CPU utilization crosses certain threshold for some amount of time.
I started testing M3 yesterday and yes I can confirm that so far that there is no issues with CPU utilization. I was about to announce to the team that an upgrade to M3 is needed. One thing Im not sure of is that I'm still seeing more Python processes in the taskmgr than what I have deployed:
Not sure if this normal and if @MattWho or @pvillard have anything to say about this.
Finally you also saved me is the time to trying to figure out why M3 is working and if this is a known issue or not so that I can post my results to the community in case someone else runs into this issue. God knows how much time I would have spent on this. Hopefully this post is enough to help others avoid the headache I had to go through.
Thanks
Dear @steven-matison ,
Let me say first you are AWESOME for going as far as you did to help me figure this thing out. I cant thank you enough.
I think my lesson learned here is that I should not rely only the release notes to see if a major issue like this has been addressed and probably always review the bug fixes under Jira to help me decide wither an upgrade is worth it or not. However having said that , I still believe something as critical as memory leak and high CPU utilization should have been mentioned as part of the highlights: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3
I really wish I knew this thing earlier, it would have saved me days of troubleshooting and the stress of being under the watch of IT because they get a notification every time CPU utilization crosses certain threshold for some amount of time.
I started testing M3 yesterday and yes I can confirm that so far that there is no issues with CPU utilization. I was about to announce to the team that an upgrade to M3 is needed. One thing Im not sure of is that I'm still seeing more Python processes in the taskmgr than what I have deployed:
Not sure if this normal and if @MattWho or @pvillard have anything to say about this.
Finally you also saved me is the time to trying to figure out why M3 is working and if this is a known issue or not so that I can post my results to the community in case someone else runs into this issue. God knows how much time I would have spent on this. Hopefully this post is enough to help others avoid the headache I had to go through.
Thanks
