[Mayday mayday] Apache NiFi Unable to create Provenance Repository

[#1]

I keep getting below error post a NiFi crash with this error [NiFi logging handler] org.apache.nifi.StdErr OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f58338b0000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)

and any attempt to restart NiFi is failing (reduce the Xms & Xmx, changed nifi.flowcontroller.autoResumeState=false, disk space is looking OK etc). Please help at the earliest -

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to create Provenance Repository at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[na:na] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[na:na] at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextDestroyed(ApplicationStartupContextListener.java:103) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:845) ~[na:na] at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:546) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:826) ~[na:na] at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:356) ~[na:na] at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1410) ~[na:na] at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1374) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:874) ~[na:na] at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:272) ~[na:na] at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:544) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) ~[na:na] at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) ~[na:na] at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) ~[na:na] at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) ~[na:na] at org.eclipse.jetty.server.Server.doStop(Server.java:482) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.apache.nifi.web.server.JettyServer.stop(JettyServer.java:854) ~[na:na] at org.apache.nifi.NiFi.shutdownHook(NiFi.java:187) [nifi-runtime-1.1.0.jar:1.1.0] at org.apache.nifi.NiFi$2.run(NiFi.java:88) [nifi-runtime-1.1.0.jar:1.1.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to create Provenance Repository at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[na:na] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[na:na] at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na] ... 33 common frames omitted Caused by: java.lang.RuntimeException: Unable to create Provenance Repository at org.apache.nifi.controller.FlowController.<init>(FlowController.java:459) ~[na:na] at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375) ~[na:na] at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na] ... 40 common frames omitted Caused by: java.lang.NullPointerException: null at org.apache.nifi.provenance.schema.EventRecord.getEvent(EventRecord.java:122) ~[na:na] at org.apache.nifi.provenance.ByteArraySchemaRecordReader.nextRecord(ByteArraySchemaRecordReader.java:77) ~[na:na] at org.apache.nifi.provenance.serialization.CompressableRecordReader.nextRecord(CompressableRecordReader.java:272) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1826) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1505) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:665) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:268) ~[na:na] at org.apache.nifi.controller.FlowController.<init>(FlowController.java:457) ~[na:na] ... 43 common frames omitted

[#2]

@Matt Burgess @Bryan Bende any help would be highly appreciated. Thank you.

[#3 · Accepted solution]

This looks like this issue:

https://issues.apache.org/jira/browse/NIFI-3186

Can you upgrade to 1.1.1?

[#4]

Thank you @Bryan Bende for such a quick response, truly appreciate it.

However this is on production and restarting NiFi is priority, even if it means losing provenance data. Please advice? Thank you.

[#5]

You should be able to delete all the files underneath "provenance_repository/journals/" and I think it will come back up correctly and you will only lose a couple of minutes or less of provenance data.

Of course if that doesn't work you can completely delete provenance_repository all together and restart with a clean provenance repo.

[#6]

Awesome, thats what I thought per this thread but wanted to be 100% sure of. Thank you @Bryan Bende.