We have a Nifi cluster that 5 days ago hung - the flow.xml.gz file got out of sync between the nodes due to prolonged connectivity issue. Ambari was also reporting a 98% JVM memory utilization (we have 32 Gigs).
Once we resync'ed the flow.xml.gz file and restarted the nodes, the JVM memory dropped back down to 5%. Since then, the JVM memory has slowly climbed back up to 82% - which Ambari is warning with a color-coded orange.
We don't have 'alot' of flows nor queued flowfiles (~300 flowfiles of less than 10k size), so I'm puzzled at the high JVM memory utilization. Any help diagnosing this issue would be greatly appreciated.
Thank you for your detailed response, Matt @Matt Clarke.
Yes, we are using HDF-18.104.22.168 and it is secured and using Ranger for NiFi authorizations.
I see that this issue is resolved in Nifi 1.6.0. Is the recommended short term fix to upgrade Nifi?
Apache NiFi 1.6 has not been released yet. It also will not include with it the Ranger plugin stuff that is included with the HDF NiFi releases versions.
If you have a support contract with Hortonworks for HDF, you could requests a hotfix build from them.
The HDF NiFi releases are never identical to any Apache NiFi release. Some processors are modified to support HDP better and some additional libraries are added (such as Ranger).
Just as a note, you can always think of the HDF releases as being build on Apache NiFi <version> plus more.