Hi, I have a 3 node Nifi Cluster running with and external Zookeeper with nifi.web.http.port=8081. and nifi.web.http.host=<IP of the box>. The cluster runs fine but suddenly goes down with the below error: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8081 On restarting it again starts running fine. My /etc/hosts file contains: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 Can someone help? @Bryan Bende ... View more

Hi, I am running nifi registry to version my nifi cluster. Things run fine, but sometimes the nifi-registry server crashes suddenly, with the below log. No changes were made prior to this crash. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} org.apache.nifi.registry.flow.FlowPersistenceException: Failed to get content of Flow ID 8ebab685-94c1-4058-ac15-439a066977dd version 3 in bucket Nifi_Cluster_Versioning:9bc3aadc-a3cd-4d2c-8ea2-a554a709c106 due to org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21. at org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:182) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.service.RegistryService.getVersionedFlowSnapshot(RegistryService.java:688) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.service.RegistryService.getFlowSnapshot(RegistryService.java:673) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.service.RegistryService$$FastClassBySpringCGLIB$$274de08.invoke(<generated>) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$581/1764534737.proceedWithInvocation(Unknown Source) ~[na:na] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.apache.nifi.registry.service.RegistryService$$EnhancerBySpringCGLIB$$71a50eee.getFlowSnapshot(<generated>) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.web.api.BucketFlowResource.getFlowVersion(BucketFlowResource.java:460) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$506/1357927175.invoke(Unknown Source) ~[na:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [jersey-common-2.26.jar:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [jersey-common-2.26.jar:na] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [jersey-server-2.26.jar:na] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [jersey-server-2.26.jar:na] at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:409) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:584) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:525) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:462) [jersey-container-servlet-core-2.26.jar:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.apache.nifi.registry.web.security.authorization.ResourceAuthorizationFilter.forwardRequestWithoutAuthorizationCheck(ResourceAuthorizationFilter.java:151) [classes/:na] at org.apache.nifi.registry.web.security.authorization.ResourceAuthorizationFilter.doFilter(ResourceAuthorizationFilter.java:113) [classes/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.apache.nifi.registry.web.security.authentication.IdentityFilter.doFilter(IdentityFilter.java:56) [classes/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.apache.nifi.registry.web.security.authentication.IdentityFilter.doFilter(IdentityFilter.java:56) [classes/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21 at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:169) ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r] at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236) ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r] at org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.getContent(GitFlowMetaData.java:423) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:180) ~[nifi-registry-framework-0.3.0.jar:0.3.0] ... 111 common frames omitted 2018-12-04 10:29:34,520 ERROR [NiFi Registry Web Server-16] o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: org.apache.nifi.registry.flow.FlowPersistenceException: Failed to get content of Flow ID 8ebab685-94c1-4058-ac15-439a066977dd version 3 in bucket Nifi_Cluster_Versioning:9bc3aadc-a3cd-4d2c-8ea2-a554a709c106 due to org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21.. Returning Internal Server Error response. ... View more

Hi, I have a clustered nifi setup and we are running GetMongo processor with the Primary mode on, so that duplicate data is not fetched. This seems to be working fine. However once I have this data I want the following processes in the chain to run on a cluster, as in parallel processing to be done on this data which has been fetched. Somehow this is not happening. So my question is below assuming GetMongo has fetched 30000 records and they are in the queue: 1) How do I check whether a processor is running its process on a single node or on all nodes. The config has been set to all nodes, but when the processor is running I see it displays 1 in the top right corner. 2) If one processor has been set to run only on primary node, do all other processors in the flow also run on Primary mode? Example: nifi.png In the screenshot above, my getmongo is running in primary node, how do I make sure that the execute script processor runs in parallel on all 3 nifi nodes. As of now if I check the view status history in the executescript process I see data flowing only through the primary node. ... View more

Hi, I have a running cluster with 3 nifi nodes, all running an embedded zookeeper. Now I want to add a fourth node which doesn't run an embedded zookeeper. What I did was, on the new node edited the nifi properties, changed nifi.cluster.is.node to true, and added the first 3 nodes zookeeper url to nifi.zookeeper.connect.string. Once done I started nifi, however I get the below error. Posting the full stack trace, this stacktrace keeps on coming infinitely: 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Below is a longer stack-trace right from startup till the error: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures} 2018-11-05 06:09:41,961 INFO [main] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317 2018-11-05 06:09:44,086 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1763ms 2018-11-05 06:09:44,247 INFO [main] org.eclipse.jetty.server.session DefaultSessionIdManager workerName=node0 2018-11-05 06:09:44,248 INFO [main] org.eclipse.jetty.server.session No SessionScavenger set, using defaults 2018-11-05 06:09:44,249 INFO [main] org.eclipse.jetty.server.session Scavenging every 660000ms 2018-11-05 06:09:44,293 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5b49b1df{/nifi-update-attribute-ui-1.7.1,file:///nifi/work/jetty/nifi-update-attribute-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.7.1.war} 2018-11-05 06:09:44,915 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=511ms 2018-11-05 06:09:44,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@11ec2b2f{/nifi-standard-content-viewer-1.7.1,file:///nifi/work/jetty/nifi-standard-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.7.1.war} 2018-11-05 06:09:45,009 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=34ms 2018-11-05 06:09:45,041 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2446be5f{/nifi-image-viewer-1.7.1,file:///nifi/work/jetty/nifi-image-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-media-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.7.1.war} 2018-11-05 06:09:46,731 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1430ms 2018-11-05 06:09:46,770 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@71dfca65{/nifi-jolt-transform-json-ui-1.7.1,file:///nifi/work/jetty/nifi-jolt-transform-json-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.7.1.war} 2018-11-05 06:09:46,959 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=38ms 2018-11-05 06:09:46,965 INFO [main] org.eclipse.jetty.ContextHandler./nifi No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:09:47,006 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d6dc2b8{/nifi,file:///nifi/work/jetty/nifi-web-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.7.1.war} 2018-11-05 06:09:47,124 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=57ms 2018-11-05 06:09:47,164 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:09:47,180 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api Initializing Spring root WebApplicationContext 2018-11-05 06:09:49,601 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties' 2018-11-05 06:09:49,602 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties' 2018-11-05 06:09:49,603 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 148 properties from /nifi/./conf/nifi.properties 2018-11-05 06:09:51,129 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.Building now... 2018-11-05 06:09:51,437 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 94 properties from system properties and environment variables 2018-11-05 06:09:51,438 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 94 properties.Including precedence overrides effective accessible registry key size is 94 2018-11-05 06:09:51,485 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions 2018-11-05 06:09:51,507 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records 2018-11-05 06:09:51,527 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2018-11-05 06:09:51,533 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 5424567582 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2018-11-05 06:09:51,534 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2018-11-05 06:09:51,657 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures} 2018-11-05 06:09:51,671 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 checkpointed with 0 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1 2018-11-05 06:09:51,755 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2018-11-05 06:09:51,826 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2018-11-05 06:10:00,725 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected 2018-11-05 06:10:00,727 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2018-11-05 06:10:01,832 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2018-11-05 06:10:01,835 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2018-11-05 06:10:01,836 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2018-11-05 06:10:01,845 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2018-11-05 06:10:04,629 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6149184e{/nifi-api,file:///nifi/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.7.1.war} 2018-11-05 06:10:05,255 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=463ms 2018-11-05 06:10:05,287 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:10:05,290 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@8eb1297{/nifi-content-viewer,file:///nifi/work/jetty/nifi-web-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.7.1.war} 2018-11-05 06:10:05,323 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=20ms 2018-11-05 06:10:05,325 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default 2018-11-05 06:10:05,367 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:10:05,370 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@646ba7be{/nifi-docs,file:///nifi/work/jetty/nifi-web-docs-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.7.1.war} 2018-11-05 06:10:05,421 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=35ms 2018-11-05 06:10:05,471 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:10:05,475 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1b6dab22{/,file:///nifi/work/jetty/nifi-web-error-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.7.1.war} 2018-11-05 06:10:05,492 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2a654167{HTTP/1.1,[http/1.1]}{<node-ip>:8081} 2018-11-05 06:10:05,492 INFO [main] org.eclipse.jetty.server.Server Started @47964ms 2018-11-05 06:10:05,505 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow... 2018-11-05 06:10:05,509 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8082 2018-11-05 06:10:05,587 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow 2018-11-05 06:10:05,610 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node:<node-ip>:8081 2018-11-05 06:10:13,959 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2018-11-05 06:10:13,959 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2018-11-05 06:10:21,670 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2018-11-05 06:10:21,670 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2018-11-05 06:10:21,675 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2018-11-05 06:10:21,677 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1d2e0826 Connection State changed to SUSPENDED p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-11-05 06:10:21,687 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more

Hi, We have a cluster running with 6 nodes. Now when I add a Kafka consumer, each cluster node should pull unique data, as in each node should fetch from a diff partition: https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka. The same is also mentioned in the nifi docs. However in our case each node is pulling the same data from Kafka leading to duplication. Can you please help. Are there any specific configurations required to get the same done? ... View more

Hi, We have a 3 node nifi cluster running, using embedded Zookeeper. Now I need to add a new node to the cluster. What is the procedure for the same? Will I have to bring down all the nodes, since I need to add an entry to the /nifi/conf/zookeeper.conf(server=) and also to the list of zookeeper servers in /nifi/conf/nifi.properties(nifi.zookeeper.connect.string). As far as I understand, these both will have to be edited on the existing 3 as well as the new box and then the nifi server restarted. But dont think this is the right way, since each time the cluster will have to be brought down. Can you please advice? ... View more