NIFI: How to turn on GC logging for NiFi?

[#1]

Hi

We are having difficulties with cluster nodes periodically becoming disconnected from the cluster. The kind of errors we are seeing in the logs are the same as the ones mentioned in this question: https://community.hortonworks.com/questions/93694/nifi-10-node-suddenly-disconnected-and-connected.h...

The suggestion in that question is that the problem is related to problems communication with Zookeeper (which is running fine in our system) and could possibly be related to Garbage Collection within NiFi causing an interruption to processing.

I would therefore like to gather some evidence as to how long GC is taking, but I can't see any GC information in the NiFi logs.

Can you tell me if it's possible to switch on logging of garbage collection within NiFi, and if not, suggest a way of monitoring GC, bearing in mind that it would not be possible for me to download and install some third-party monitoring tool on our system.

Thanks

Richard

[#2 · Accepted solution]

@Richard Corfield

You can get GC output logs out of NiFi by adding the following lines to NiFi's bootstrap.conf file:

java.arg.20=-XX:+PrintGCDetails

java.arg.21=-XX:+PrintGCTimeStamps

java.arg.22=-XX:+PrintGCDateStamps

java.arg.23=-Xloggc:<file>

The last entry allows you to specific a separate log file for this output to be written in to rather then stdout.

Thanks,

Matt

[#3]

Thank you. I've tested this on our Dev system and it looks like it's exactly what we need.

Regards

Richard

[#4]

Thank you for your solution. It works wonders.

In the meantime I don't think there is a default housekeeping process.

Do you know how to control the rollover of the GC logs?

Thank you in advance

[#5]

@DavidR 

Typically you would only enable GC debug logging to investigate and issue and not leave these properties in all the time.

If you intend on leaving these in all the time, i suggest removing the following line:

java.arg.23=-Xloggc:<file>

With no log file defined the GC logs will go directly in to the nifi-bootstrap.log.

You can then easily control/configure log rotation and retention for the NiFi bootstrap log in the logback.xml file found in the NiFi conf directory.

Thanks,

Matt 

[#6]

@MattWho 

Thank you for your input. It works and It was invaluable.