Nifi often restarts automatically, causing processor error

[#1]

I have a processor group that fetches data from ElasticSearch and stores it in mysql. Every time I run this processor group, NiFi will restart automatically after a while, which will cause my 'Scroll Duration=2m' to be disabled in ScrollElasticSearchHttp,and then I will never get the new data from the ES.

I checked the log file and found no error message.

this is the nifi-bootstrap_2018-10-09.log

2018-10-09 09:43:15,167 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache NiFi appears to have died. Restarting...
2018-10-09 09:43:15,343 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 15861
2018-10-09 09:43:15,373 INFO [main] org.apache.nifi.bootstrap.RunNiFi Successfully started Apache NiFi with PID 15861
2018-10-09 09:43:16,401 INFO [NiFi logging handler] org.apache.nifi.StdOut Listening for transport dt_socket at address: 8111
2018-10-09 09:43:21,932 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 33718

this is part of nifi-app_2018-10-09_09.0.log

2018-10-09 09:41:51,464 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-10-09 09:41:56,114 INFO [FileSystemRepository Workers Thread-2] o.a.n.c.repository.FileSystemRepository Successfully archived 24 Resource Claims for Container default in 35 millis
2018-10-09 09:41:57,747 INFO [pool-10-thread-1] o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 2478 Records and 0 Swap Files in 5959 milliseconds (Stop-the-world time = 5475 milliseconds), max Transaction ID 28856764
2018-10-09 09:41:57,747 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2478 records in 5959 milliseconds
2018-10-09 09:42:01,363 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 1698404
2018-10-09 09:42:10,417 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for ./provenance_repository/index-1539049279000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-10-09 09:42:10,471 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (54600 records) into single Provenance Log File ./provenance_repository/1595821.prov in 51243 milliseconds
2018-10-09 09:42:10,471 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 32324 records. In the past 5 minutes, 450054 events have been written to the Provenance Repository, totaling 466.27 MB
2018-10-09 09:42:11,069 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@52f1d01b checkpointed with 11 Records and 0 Swap Files in 1365 milliseconds (Stop-the-world time = 628 milliseconds, Clear Edit Logs time = 677 millis), max Transaction ID 4721
2018-10-09 09:42:17,773 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for ./provenance_repository/index-1539049279000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-10-09 09:42:17,783 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (47983 records) into single Provenance Log File ./provenance_repository/1650421.prov in 7273 milliseconds
2018-10-09 09:42:17,784 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 28986 records. In the past 5 minutes, 468675 events have been written to the Provenance Repository, totaling 483.25 MB
2018-10-09 09:42:41,503 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 1775436
2018-10-09 09:42:54,028 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for ./provenance_repository/index-1539049279000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-10-09 09:42:54,112 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (77032 records) into single Provenance Log File ./provenance_repository/1698404.prov in 12684 milliseconds
2018-10-09 09:42:54,112 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 46170 records. In the past 5 minutes, 469450 events have been written to the Provenance Repository, totaling 486.88 MB
2018-10-09 09:43:20,130 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-09 09:43:21,859 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./conf/nifi.properties'
2018-10-09 09:43:21,884 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 154 properties from /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./conf/nifi.properties
2018-10-09 09:43:21,904 INFO [main] org.apache.nifi.NiFi Loaded 154 properties
2018-10-09 09:43:21,923 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 33718
2018-10-09 09:43:21,933 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2018-10-09 09:43:24,367 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 97 NAR files with all processors...
2018-10-09 09:43:47,958 INFO [main] org.apache.nifi.nar.NarUnpacker NAR loading process took 23591440189 nanoseconds (23 seconds).
2018-10-09 09:45:20,709 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-jetty-bundle-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-jetty-bundle-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,740 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/framework/nifi-framework-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,743 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-kerberos-iaa-providers-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-kerberos-iaa-providers-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,743 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-cybersecurity-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-cybersecurity-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,746 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-datadog-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-datadog-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,748 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-poi-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-poi-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,758 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-media-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-media-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,760 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-avro-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-avro-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,762 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-ambari-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-ambari-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,763 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-spring-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-spring-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,764 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-slack-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-slack-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,766 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-enrich-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-enrich-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,769 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-provenance-repository-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-provenance-repository-nar-1.7.2-SNAPSHOT.nar-unpacked]

I want to know why the NiFi process will die and there is no error message. How should I solve this problem?

[#2 · Accepted solution]

@Cooper Max

NiFi has two processes as you see above that are running. The NiFi bootstrap process is what is kicked off when starting NiFi and it then spans off the main NiFi process. The bootstrap process then monitors for the pid of that main process and if it disappears, the output you see above is thrown and the bootstrap then attempts tp restart the main process.

-

If your nifi-app.log is not exhibiting any signs of issues going on in your dataflow leading up to this event, the killing of this NiFi process is being triggered external to NiFi.

-

Most commonly you may find that the server itself has killed the process. I would suggest looking at your server logs for the execution of "OOM killer". When memory resources on a server reach usage a level where the OS feels it could result in the server become unresponsive or crash, oom killer is launched which evaluates the running process and elects a process to be killed to free memory to protect the OS. Considering the memory footprint of a typical main NiFi JVM process, it is commonly selected by the oom killer.

-

To resolve this issue, you would need to reduce the amount of memory that is being consumed by running process on this same server.
- Do not run NIFi on server where other service are co-located

- Reduce the configured JVM setting for the NiFi process in the nifi-bootsrap.conf file.

----- Above may require you to re-evaluate your dataflow design(s) in NiFi to reduce heap memory usage.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

[#3]

Hi Matt.

Thank you, it is indeed the server is out of memory, I have found it in the system log.