Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI: How to turn on GC logging for NiFi?

Solved Go to solution

NIFI: How to turn on GC logging for NiFi?

New Contributor

Hi

We are having difficulties with cluster nodes periodically becoming disconnected from the cluster. The kind of errors we are seeing in the logs are the same as the ones mentioned in this question: https://community.hortonworks.com/questions/93694/nifi-10-node-suddenly-disconnected-and-connected.h...

The suggestion in that question is that the problem is related to problems communication with Zookeeper (which is running fine in our system) and could possibly be related to Garbage Collection within NiFi causing an interruption to processing.

I would therefore like to gather some evidence as to how long GC is taking, but I can't see any GC information in the NiFi logs.

Can you tell me if it's possible to switch on logging of garbage collection within NiFi, and if not, suggest a way of monitoring GC, bearing in mind that it would not be possible for me to download and install some third-party monitoring tool on our system.

Thanks

Richard

1 ACCEPTED SOLUTION

Accepted Solutions

Re: NIFI: How to turn on GC logging for NiFi?

Master Guru

@Richard Corfield

You can get GC output logs out of NiFi by adding the following lines to NiFi's bootstrap.conf file:

java.arg.20=-XX:+PrintGCDetails

java.arg.21=-XX:+PrintGCTimeStamps

java.arg.22=-XX:+PrintGCDateStamps

java.arg.23=-Xloggc:<file>

The last entry allows you to specific a separate log file for this output to be written in to rather then stdout.

Thanks,

Matt

5 REPLIES 5

Re: NIFI: How to turn on GC logging for NiFi?

Master Guru

@Richard Corfield

You can get GC output logs out of NiFi by adding the following lines to NiFi's bootstrap.conf file:

java.arg.20=-XX:+PrintGCDetails

java.arg.21=-XX:+PrintGCTimeStamps

java.arg.22=-XX:+PrintGCDateStamps

java.arg.23=-Xloggc:<file>

The last entry allows you to specific a separate log file for this output to be written in to rather then stdout.

Thanks,

Matt

Re: NIFI: How to turn on GC logging for NiFi?

New Contributor

Thank you. I've tested this on our Dev system and it looks like it's exactly what we need.

Regards

Richard

Re: NIFI: How to turn on GC logging for NiFi?

New Contributor

Thank you for your solution. It works wonders.

In the meantime I don't think there is a default housekeeping process.

Do you know how to control the rollover of the GC logs?

 

Thank you in advance

Highlighted

Re: NIFI: How to turn on GC logging for NiFi?

Master Guru

@DavidR 

 

Typically you would only enable GC debug logging to investigate and issue and not leave these properties in all the time.

If you intend on leaving these in all the time, i suggest removing the following line:

 

java.arg.23=-Xloggc:<file>

 

 

With no log file defined the GC logs will go directly in to the nifi-bootstrap.log.

You can then easily control/configure log rotation and retention for the NiFi bootstrap log in the logback.xml file found in the NiFi conf directory.

 

Thanks,

Matt 

Re: NIFI: How to turn on GC logging for NiFi?

New Contributor

@MattWho 

Thank you for your input. It works and It was invaluable. 

 

Don't have an account?
Coming from Hortonworks? Activate your account here