Support Questions
Find answers, ask questions, and share your expertise

[Mayday mayday] Apache NiFi Unable to create Provenance Repository

Contributor

I keep getting below error post a NiFi crash with this error [NiFi logging handler] org.apache.nifi.StdErr OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f58338b0000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)

and any attempt to restart NiFi is failing (reduce the Xms & Xmx, changed nifi.flowcontroller.autoResumeState=false, disk space is looking OK etc). Please help at the earliest -

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to create Provenance Repository at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[na:na] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[na:na] at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextDestroyed(ApplicationStartupContextListener.java:103) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:845) ~[na:na] at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:546) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:826) ~[na:na] at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:356) ~[na:na] at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1410) ~[na:na] at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1374) ~[na:na] at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:874) ~[na:na] at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:272) ~[na:na] at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:544) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) ~[na:na] at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) ~[na:na] at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) ~[na:na] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) ~[na:na] at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) ~[na:na] at org.eclipse.jetty.server.Server.doStop(Server.java:482) ~[na:na] at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) ~[na:na] at org.apache.nifi.web.server.JettyServer.stop(JettyServer.java:854) ~[na:na] at org.apache.nifi.NiFi.shutdownHook(NiFi.java:187) [nifi-runtime-1.1.0.jar:1.1.0] at org.apache.nifi.NiFi$2.run(NiFi.java:88) [nifi-runtime-1.1.0.jar:1.1.0] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to create Provenance Repository at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[na:na] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[na:na] at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na] ... 33 common frames omitted Caused by: java.lang.RuntimeException: Unable to create Provenance Repository at org.apache.nifi.controller.FlowController.<init>(FlowController.java:459) ~[na:na] at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375) ~[na:na] at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na] ... 40 common frames omitted Caused by: java.lang.NullPointerException: null at org.apache.nifi.provenance.schema.EventRecord.getEvent(EventRecord.java:122) ~[na:na] at org.apache.nifi.provenance.ByteArraySchemaRecordReader.nextRecord(ByteArraySchemaRecordReader.java:77) ~[na:na] at org.apache.nifi.provenance.serialization.CompressableRecordReader.nextRecord(CompressableRecordReader.java:272) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1826) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1505) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:665) ~[na:na] at org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:268) ~[na:na] at org.apache.nifi.controller.FlowController.<init>(FlowController.java:457) ~[na:na] ... 43 common frames omitted

1 ACCEPTED SOLUTION

Accepted Solutions

This looks like this issue:

https://issues.apache.org/jira/browse/NIFI-3186

Can you upgrade to 1.1.1?

View solution in original post

5 REPLIES 5

Contributor

@Matt Burgess @Bryan Bende any help would be highly appreciated. Thank you.

This looks like this issue:

https://issues.apache.org/jira/browse/NIFI-3186

Can you upgrade to 1.1.1?

View solution in original post

Contributor

Thank you @Bryan Bende for such a quick response, truly appreciate it.

However this is on production and restarting NiFi is priority, even if it means losing provenance data. Please advice? Thank you.

You should be able to delete all the files underneath "provenance_repository/journals/" and I think it will come back up correctly and you will only lose a couple of minutes or less of provenance data.

Of course if that doesn't work you can completely delete provenance_repository all together and restart with a clean provenance repo.

Contributor

Awesome, thats what I thought per this thread but wanted to be 100% sure of. Thank you @Bryan Bende.