@Kilynn I see that you have set some very high values (330 GB) for your NiFi heap space. This is really not recommended. Can also see from what you have shared that the particular NiFi instance has been up for ~2.5 hours and within that 2.5 hour timeframe it has been in a stop-the-world state due to Java Garbage Collection (GC) for ~20 minutes. This means that your NiFi is spending ~13% of its uptime doing nothing by stop-the-world garbage collection. Is there a reason you set your heap this high? Setting large heap simply because your server has the memory available is not a good reason. Java GC kicks in when heap usage is around 80% usage. When an object in heap is no longer being used it is not actually cleaned out of heap and space reclaimed at that time. Space is only reclaimed via GC. The larger the heap the longer the GC events are going to take. Seeing as how you current heap usage is ~53%, I am guessing with each GC event a considerable amount of heap space is being released. Long GC pauses (stop-the-world) can result in node disconnection because NiFi node heartbeats are not being sent. When NiFi seems to be restarting with no indication at all in the nifi-app.log, you may want to take a look at your systems /var/log/messages file for any indications of the Out Of Memory (OOM) killer being executed. With your NiFi instance being the largest consumer of memory on the host, it would be the top pick process by the OS OOM Killer when available OS memory gets too low. In most case you should not need more the 16GB of heap to run most dataflows. If you are building dataflows that utilize a lot of heap, I'd recommend taking a close look at your dataflow designs to see if there are better design choices. This that can lead to large heap usage is creating very large FlowFile attributes (for example, extracting large amount of a FlowFile's content in to a FlowFile attribute). FlowFile attributes all live inside the JVM heap. Some other dataflow designs elements that can lead to high heap usage include: - Merging a very large number of FlowFiles in a single Merge processor. Better to use multiple merge processors in series with first merging up 10,000 to 20,000 FlowFiles and then second merging those in to even larger files - Splitting a very large file in a single split processor resulting in a lot of FlowFiles produced in a single transaction. Better to use multiple or even look at ways of processing the dat without needing to split the file in to multiple files (look at using the available "record" based processors) - Reading in entire content of a large FlowFile in to memory to perform action on it. Like ExtractText processor configured for entire text instead of line-by-line mode. While I 100% agree that you should be looking in to your "thread" allocation choices in your nifi.properties file. Many of them seem unnecessarily high for a 7 node cluster. You should understand that each node in your cluster executes independently of the others, so the settings applied pertain to each node individually. Following the guidance in the NiFi app guide (can be found in NIFi UI under help or on Apache NiFi site) is strongly advised. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
