Nifi JVM Memory Utilization Continually Rising...

[#1]

Hey Guys,

We have a Nifi cluster that 5 days ago hung - the flow.xml.gz file got out of sync between the nodes due to prolonged connectivity issue. Ambari was also reporting a 98% JVM memory utilization (we have 32 Gigs).

Once we resync'ed the flow.xml.gz file and restarted the nodes, the JVM memory dropped back down to 5%. Since then, the JVM memory has slowly climbed back up to 82% - which Ambari is warning with a color-coded orange.

We don't have 'alot' of flows nor queued flowfiles (~300 flowfiles of less than 10k size), so I'm puzzled at the high JVM memory utilization. Any help diagnosing this issue would be greatly appreciated.

[#2 · Accepted solution]

@Sonny Chee

What version of HDF are you running? Is it HDF 3.1?

Is your NiFi secured and using Ranger for NiFi authorizations?

If so, you may have run into this identified bug:

https://issues.apache.org/jira/browse/NIFI-4925

There are currently no HDF releases with the fix included.

Thank you,

Matt

[#3]

Thank you for your detailed response, Matt @Matt Clarke.

Yes, we are using HDF-3.1.0.0 and it is secured and using Ranger for NiFi authorizations.

I see that this issue is resolved in Nifi 1.6.0. Is the recommended short term fix to upgrade Nifi?

[#4]

@Sonny Chee

Apache NiFi 1.6 has not been released yet. It also will not include with it the Ranger plugin stuff that is included with the HDF NiFi releases versions.

If you have a support contract with Hortonworks for HDF, you could requests a hotfix build from them.

The HDF NiFi releases are never identical to any Apache NiFi release. Some processors are modified to support HDP better and some additional libraries are added (such as Ranger).

Just as a note, you can always think of the HDF releases as being build on Apache NiFi <version> plus more.

[#5]

Thank you for the clarification.