No problem... as soon as you added that you were running HDF 2.1.2, it helped. ... View more

@Oliver Meyn You have run in to the following bug: https://issues.apache.org/jira/browse/NIFI-3664 The good news is that the fix for this bug is part of HDF 2.1.3. Thank you, Matt ... View more

@Joshua Adeleke Are you able to add new controller services? ... View more

Your FGC is not very high at only 4. Do you still have issues if you restart your cluster with all components in a stopped state? This would at least show that your issue is dataflow related. To start with all components stopped, edit the nifi.properties file on all nodes and change the following property to "false" nifi.flowcontroller.autoResumeState=true Once you cluster is back up and you can access the UI, check for anywhere in your stopped flow where you have queued data (look at largest queues first). Start these flows first and look at impact they are having on GC and Heap memory usage. Another possibility is that you are are oversubscribing to your available resources. Use the "top" command to observer impact on your system's resources (Namely CPU). Make sure you have not allocated to many concurrent tasks to your individual processors. Make sure you have not set your "max timer driven thread count" and "max event driven thread count" values to high. These values should be set to 2-4 times the number of cores you have in one node. (fro example if you have a 4 node cluster and each node in your cluster has 16 cores, the max values set should be between 32 and 64 (max 2-4 times 16 cores). Thanks, Matt ... View more

@Xi Sanderson The above log line is not an indication of a problem. You can use the jstat command to look at your GC stats: For example the following command will print GC stats every 250 ms for 1000 lines: /<path to Java>/bin/jstat -gcutil <nifi pid> 250 1000 Column Description S0 Survivor space 0 utilization as a percentage of the space's current capacity. S1 Survivor space 1 utilization as a percentage of the space's current capacity. E Eden space utilization as a percentage of the space's current capacity. O Old space utilization as a percentage of the space's current capacity. P Permanent space utilization as a percentage of the space's current capacity. YGC Number of young generation GC events. YGCT Young generation garbage collection time. FGC Number of full GC events. FGCT Full garbage collection time. GCT Total garbage collection time. What version of NiFi and what version of Java are you running? Thanks, Matt ... View more

@Xi Sanderson The most common reason for not seeing Heartbeats generated on the configured interval is when you have a lot of java full garbage collection (GC) going on. Garbage collection is a stop the world event. The JVM will do nothing else until the garbage collection has completed. Partial garbage collection is normal and healthy and stop the world events here should be extremely short in duration. You can look at the GC stats fro your NIFi cluster via the system diagnostics UI. You can get tho this UI via link in the lower right corner of the NiFi summary UI: Young is normal to see, but you want to see hopefully 0 Old Gen events. If GC is your issue, you will want to figure out what has changed. New data types or volumes, new dataflows, etc... Thanks, Matt ... View more