Member since
10-09-2018
2
Posts
0
Kudos Received
0
Solutions
10-10-2018
01:24 AM
Hi Matt. Thank you, it is indeed the server is out of memory, I have found it in the system log.
... View more
10-09-2018
03:52 AM
I have a processor group that fetches data from ElasticSearch and stores it in mysql. Every time I run this processor group, NiFi will restart automatically after a while, which will cause my 'Scroll Duration=2m' to be disabled in ScrollElasticSearchHttp,and then I will never get the new data from the ES. I checked the log file and found no error message. this is the nifi-bootstrap_2018-10-09.log 2018-10-09 09:43:15,167 WARN [main] org.apache.nifi.bootstrap.RunNiFi Apache NiFi appears to have died. Restarting...
2018-10-09 09:43:15,343 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 15861
2018-10-09 09:43:15,373 INFO [main] org.apache.nifi.bootstrap.RunNiFi Successfully started Apache NiFi with PID 15861
2018-10-09 09:43:16,401 INFO [NiFi logging handler] org.apache.nifi.StdOut Listening for transport dt_socket at address: 8111
2018-10-09 09:43:21,932 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 33718
this is part of nifi-app_2018-10-09_09.0.log 2018-10-09 09:41:51,464 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-10-09 09:41:56,114 INFO [FileSystemRepository Workers Thread-2] o.a.n.c.repository.FileSystemRepository Successfully archived 24 Resource Claims for Container default in 35 millis
2018-10-09 09:41:57,747 INFO [pool-10-thread-1] o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 2478 Records and 0 Swap Files in 5959 milliseconds (Stop-the-world time = 5475 milliseconds), max Transaction ID 28856764
2018-10-09 09:41:57,747 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 2478 records in 5959 milliseconds
2018-10-09 09:42:01,363 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 1698404
2018-10-09 09:42:10,417 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for ./provenance_repository/index-1539049279000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-10-09 09:42:10,471 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (54600 records) into single Provenance Log File ./provenance_repository/1595821.prov in 51243 milliseconds
2018-10-09 09:42:10,471 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 32324 records. In the past 5 minutes, 450054 events have been written to the Provenance Repository, totaling 466.27 MB
2018-10-09 09:42:11,069 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@52f1d01b checkpointed with 11 Records and 0 Swap Files in 1365 milliseconds (Stop-the-world time = 628 milliseconds, Clear Edit Logs time = 677 millis), max Transaction ID 4721
2018-10-09 09:42:17,773 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for ./provenance_repository/index-1539049279000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-10-09 09:42:17,783 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (47983 records) into single Provenance Log File ./provenance_repository/1650421.prov in 7273 milliseconds
2018-10-09 09:42:17,784 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 28986 records. In the past 5 minutes, 468675 events have been written to the Provenance Repository, totaling 483.25 MB
2018-10-09 09:42:41,503 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 1775436
2018-10-09 09:42:54,028 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for ./provenance_repository/index-1539049279000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-10-09 09:42:54,112 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (77032 records) into single Provenance Log File ./provenance_repository/1698404.prov in 12684 milliseconds
2018-10-09 09:42:54,112 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 46170 records. In the past 5 minutes, 469450 events have been written to the Provenance Repository, totaling 486.88 MB
2018-10-09 09:43:20,130 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-10-09 09:43:21,859 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./conf/nifi.properties'
2018-10-09 09:43:21,884 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 154 properties from /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./conf/nifi.properties
2018-10-09 09:43:21,904 INFO [main] org.apache.nifi.NiFi Loaded 154 properties
2018-10-09 09:43:21,923 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 33718
2018-10-09 09:43:21,933 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
2018-10-09 09:43:24,367 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 97 NAR files with all processors...
2018-10-09 09:43:47,958 INFO [main] org.apache.nifi.nar.NarUnpacker NAR loading process took 23591440189 nanoseconds (23 seconds).
2018-10-09 09:45:20,709 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-jetty-bundle-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-jetty-bundle-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,740 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/framework/nifi-framework-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,743 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-kerberos-iaa-providers-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-kerberos-iaa-providers-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,743 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-cybersecurity-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-cybersecurity-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,746 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-datadog-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-datadog-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,748 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-poi-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-poi-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,758 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-media-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-media-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,760 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-avro-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-avro-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,762 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-ambari-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-ambari-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,763 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-spring-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-spring-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,764 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-slack-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-slack-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,766 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-enrich-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-enrich-nar-1.7.2-SNAPSHOT.nar-unpacked]
2018-10-09 09:45:20,769 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: /data/hdp/nifi/macun/nifi-1.7.2-SNAPSHOT/./work/nar/extensions/nifi-provenance-repository-nar-1.7.2-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-provenance-repository-nar-1.7.2-SNAPSHOT.nar-unpacked]
I want to know why the NiFi process will die and there is no error message. How should I solve this problem?
... View more
Labels:
- Labels:
-
Apache NiFi