NiFi shutting down abruptly

[#1]

NiFi services is shutting down abruptly with below exception in production environment. So i tried few things to troubleshoot, when i remove existing flow.xml.gz and restart services i see NiFi UI works as expected. But when i try using flow.xml.gz from archive (latest) and restart i get below error. Not sure what is causing issue here..Appreciate quick response as its impacting production run.

Log file nifi-app.log doesnt have any error.

Below is error from hs_err_pid*.log

Internal exceptions (10 events):
Event: 25.581 Thread 0x00007f6cef29e800 Exception <a 'java/lang/NoSuchMethodError': java.lang.Object.lambda$comparingInt$7b0bb60$1(Ljava/util/function/ToIntFunction;Ljava/lang/Object;Ljava/lang/Object;)I> (0x00000005bdf0e618) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/
Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7efdb410 to 0x00007f6e7efddb85
Event: 25.665 Thread 0x00007f6cef29e800 Implicit null exception at 0x00007f6e7d49a697 to 0x00007f6e7d49a7b5
Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc39b8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 25.784 Thread 0x00007f6ceea03800 Exception <a 'sun/nio/fs/UnixException'> (0x00000005c2dc3f60) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d148) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 25.794 Thread 0x00007f6a1c001000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005d838d6f0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 25.832 Thread 0x00007f6a64002800 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959
Event: 25.832 Thread 0x00007f6b08005000 Implicit null exception at 0x00007f6e7e330862 to 0x00007f6e7e330959
Event: 25.839 Thread 0x00007f6b08005000 Exception <a 'sun/nio/fs/UnixException'> (0x00000005db870240) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 710]

Bootstrap error log :

2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # An error report file with more information is saved as:
2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut # /users/quant_hub/nifi/nifi-1.11.1/hs_err_pid20683.log
2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140348715132672 also had an error]
2020-08-10 10:16:51,345 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349382018816 also had an error]
2020-08-10 10:16:51,393 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349099882240 also had an error]
2020-08-10 10:16:51,402 INFO [NiFi logging handler] org.apache.nifi.StdOut [thread 140349347297024 also had an error]
2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut #
2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # If you would like to submit a bug report, please visit:
2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # http://bugreport.java.com/bugreport/crash.jsp
2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # The crash happened outside the Java Virtual Machine in native code.
2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut # See problematic frame for where to report the bug.
2020-08-10 10:16:51,447 INFO [NiFi logging handler] org.apache.nifi.StdOut #
2020-08-10 10:16:59,250 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []
2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED
2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED
2020-08-10 10:16:59,255 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED
2020-08-10 10:16:59,327 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is running at PID 20683 but is not responding to ping requests
2020-08-10 10:17:10,140 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi

@dfheinz @MattWho 

[#2]

Unfortunately I am not able to help with the specific content of the error, but as you mention it is urgent and impacting a production run, I would highly recommend you to create a support ticket with the relevant information.

You may ultimately get an answer via the community, but for urgent matters logging a support ticket is really the recommended course of action.

----

It may of course be possible there is a problem with your flow, consider trying to roll back to an older version.

[#3]

@Gubbi 

Since you comment that NiFi starts fine when you remove the flow.xml.gz, this points at an issue with loading /reading the flow.xml.gz file.  I would suggest opening you flow.xml.gz in an xml editor and make sure it is valid.  When NiFi starts it loads the flow.xml in to heap memory.  From then on all changes made within the UI are made in memory and a flow.xml.gz is written to disk.  I would be looking for XML special characters (http://xml.silmaril.ie/specials.html) that were not escaped in the XML that written out to the flow.xml.gz.  Manually remove or correct the invalid flow.xml.gz may work.

You may get lucky going back and trying multiple archived copies of flow.xml.gz files.  Perhaps one still exist prior to the config change that was made that introduced the issue?

Note: Be aware that starting NiFi without the flow.xml.gz will result in loss of all queued FlowFiles.

Hope this helps,

Matt