Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi Registry Crashes/ Stops

Highlighted

Nifi Registry Crashes/ Stops

Explorer

Hi,

I am running nifi registry to version my nifi cluster. Things run fine, but sometimes the nifi-registry server crashes suddenly, with the below log. No changes were made prior to this crash.

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre}

org.apache.nifi.registry.flow.FlowPersistenceException: Failed to get content of Flow ID 8ebab685-94c1-4058-ac15-439a066977dd version 3 in bucket Nifi_Cluster_Versioning:9bc3aadc-a3cd-4d2c-8ea2-a554a709c106 due to org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21.

at org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:182) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

at org.apache.nifi.registry.service.RegistryService.getVersionedFlowSnapshot(RegistryService.java:688) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

at org.apache.nifi.registry.service.RegistryService.getFlowSnapshot(RegistryService.java:673) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

at org.apache.nifi.registry.service.RegistryService$$FastClassBySpringCGLIB$$274de08.invoke(<generated>) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$581/1764534737.proceedWithInvocation(Unknown Source) ~[na:na]

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.apache.nifi.registry.service.RegistryService$$EnhancerBySpringCGLIB$$71a50eee.getFlowSnapshot(<generated>) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

at org.apache.nifi.registry.web.api.BucketFlowResource.getFlowVersion(BucketFlowResource.java:460) ~[classes/:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]

at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$506/1357927175.invoke(Unknown Source) ~[na:na]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ~[jersey-server-2.26.jar:na]

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [jersey-common-2.26.jar:na]

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [jersey-common-2.26.jar:na]

at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [jersey-common-2.26.jar:na]

at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [jersey-common-2.26.jar:na]

at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [jersey-common-2.26.jar:na]

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [jersey-common-2.26.jar:na]

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [jersey-server-2.26.jar:na]

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [jersey-server-2.26.jar:na]

at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [jersey-container-servlet-core-2.26.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:409) [jersey-container-servlet-core-2.26.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:584) [jersey-container-servlet-core-2.26.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:525) [jersey-container-servlet-core-2.26.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:462) [jersey-container-servlet-core-2.26.jar:na]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.apache.nifi.registry.web.security.authorization.ResourceAuthorizationFilter.forwardRequestWithoutAuthorizationCheck(ResourceAuthorizationFilter.java:151) [classes/:na]

at org.apache.nifi.registry.web.security.authorization.ResourceAuthorizationFilter.doFilter(ResourceAuthorizationFilter.java:113) [classes/:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.apache.nifi.registry.web.security.authentication.IdentityFilter.doFilter(IdentityFilter.java:56) [classes/:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.apache.nifi.registry.web.security.authentication.IdentityFilter.doFilter(IdentityFilter.java:56) [classes/:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]

at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]

at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21

at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:169) ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r]

at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236) ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r]

at org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.getContent(GitFlowMetaData.java:423) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

at org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:180) ~[nifi-registry-framework-0.3.0.jar:0.3.0]

... 111 common frames omitted

2018-12-04 10:29:34,520 ERROR [NiFi Registry Web Server-16] o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: org.apache.nifi.registry.flow.FlowPersistenceException: Failed to get content of Flow ID 8ebab685-94c1-4058-ac15-439a066977dd version 3 in bucket Nifi_Cluster_Versioning:9bc3aadc-a3cd-4d2c-8ea2-a554a709c106 due to org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21.. Returning Internal Server Error response.

2 REPLIES 2
Highlighted

Re: Nifi Registry Crashes/ Stops

Is there any pattern you can find to reproduce this?

If you restart NiFi Registry after this happens, is it working ok again, or does it have an issue on start up?

Re: Nifi Registry Crashes/ Stops

Super Guru

I have seen this when git wasn't available, network issues or if someone changed outside of the versioning.

what version of NiFi? NiFi Registry? Java?

Don't have an account?
Coming from Hortonworks? Activate your account here