Member since
10-30-2018
26
Posts
2
Kudos Received
0
Solutions
04-23-2019
08:53 AM
@Geoffrey Shelton Okot In that case why would it work fine for sometime before going down? Also if I comment out 127.0.0.1, the nifi server doesn't start at all.
... View more
04-18-2019
08:11 AM
Hi, I have a 3 node Nifi Cluster running with and external Zookeeper with nifi.web.http.port=8081. and nifi.web.http.host=<IP of the box>. The cluster runs fine but suddenly goes down with the below error: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8081 On restarting it again starts running fine. My /etc/hosts file contains: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 Can someone help? @Bryan Bende
... View more
Labels:
- Labels:
-
Apache NiFi
02-21-2019
11:09 AM
Hi @Kamal Sehgal: What is the issue that you are facing. Also I found this useful: https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/ https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/ Its actually easier using an external zookeeper(first link has the steps) . In case you get stuck, let me know. Also there is no concept of manager node in nifi now. Refer to, https://hortonworks.com/blog/apache-nifi-1-0-0-zero-master-clustering/
... View more
12-05-2018
04:38 AM
Hi, I am running nifi registry to version my nifi cluster. Things run fine, but sometimes the nifi-registry server crashes suddenly, with the below log. No changes were made prior to this crash. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.Apple-tab-span {white-space:pre} org.apache.nifi.registry.flow.FlowPersistenceException: Failed to get content of Flow ID 8ebab685-94c1-4058-ac15-439a066977dd version 3 in bucket Nifi_Cluster_Versioning:9bc3aadc-a3cd-4d2c-8ea2-a554a709c106 due to org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21. at org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:182) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.service.RegistryService.getVersionedFlowSnapshot(RegistryService.java:688) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.service.RegistryService.getFlowSnapshot(RegistryService.java:673) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.service.RegistryService$$FastClassBySpringCGLIB$$274de08.invoke(<generated>) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$581/1764534737.proceedWithInvocation(Unknown Source) ~[na:na] at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.apache.nifi.registry.service.RegistryService$$EnhancerBySpringCGLIB$$71a50eee.getFlowSnapshot(<generated>) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.web.api.BucketFlowResource.getFlowVersion(BucketFlowResource.java:460) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$506/1357927175.invoke(Unknown Source) ~[na:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) ~[jersey-server-2.26.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [jersey-common-2.26.jar:na] at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [jersey-common-2.26.jar:na] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [jersey-common-2.26.jar:na] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [jersey-server-2.26.jar:na] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [jersey-server-2.26.jar:na] at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:409) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:584) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:525) [jersey-container-servlet-core-2.26.jar:na] at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:462) [jersey-container-servlet-core-2.26.jar:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.apache.nifi.registry.web.security.authorization.ResourceAuthorizationFilter.forwardRequestWithoutAuthorizationCheck(ResourceAuthorizationFilter.java:151) [classes/:na] at org.apache.nifi.registry.web.security.authorization.ResourceAuthorizationFilter.doFilter(ResourceAuthorizationFilter.java:113) [classes/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.apache.nifi.registry.web.security.authentication.IdentityFilter.doFilter(IdentityFilter.java:56) [classes/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.apache.nifi.registry.web.security.authentication.IdentityFilter.doFilter(IdentityFilter.java:56) [classes/:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.0.7.RELEASE.jar:5.0.7.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:105) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:123) [spring-boot-2.0.4.RELEASE.jar:2.0.4.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar:5.0.8.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [jetty-servlet-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.Server.handle(Server.java:531) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) [jetty-util-9.4.11.v20180605.jar:9.4.11.v20180605] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21 at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:169) ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r] at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236) ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r] at org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.getContent(GitFlowMetaData.java:423) ~[nifi-registry-framework-0.3.0.jar:0.3.0] at org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:180) ~[nifi-registry-framework-0.3.0.jar:0.3.0] ... 111 common frames omitted 2018-12-04 10:29:34,520 ERROR [NiFi Registry Web Server-16] o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: org.apache.nifi.registry.flow.FlowPersistenceException: Failed to get content of Flow ID 8ebab685-94c1-4058-ac15-439a066977dd version 3 in bucket Nifi_Cluster_Versioning:9bc3aadc-a3cd-4d2c-8ea2-a554a709c106 due to org.eclipse.jgit.errors.MissingObjectException: Missing unknown dec0e2baa3fdfba6baf3d658c9003f611005fb21.. Returning Internal Server Error response.
... View more
Labels:
- Labels:
-
Apache NiFi
12-04-2018
10:38 AM
I tested it out on a cluster, works perfectly. Any changes made from any of the nifi cluster nodes and committed are reflected across all nifi nodes.
... View more
12-04-2018
04:30 AM
@Bryan Bende: thanks, just one more thing, does this mean that if I make a change to one node in the cluster and commit, all the nodes in the nifi cluster will show that the commit has happened?
... View more
12-03-2018
08:50 AM
Hi, I have a 3 node nifi cluster. I want to persist my flow file changes to a Git repo. I could do it locally on my single node system and it is very helpful in versioning. Wanted to check how will the configuration be done in the 3 node cluster. Can I just have registry on only one node in the cluster or should I put it in all 3 nodes. Also if such a thing is not supported, is there any work around to use nifi registry with a nifi cluster.
... View more
Labels:
- Labels:
-
Apache NiFi
11-23-2018
04:37 PM
1 Kudo
That is true, but some times such an option can be good for a little access control and avoiding not seriously harmfull but unwanted activities on the server.
... View more
11-16-2018
11:03 AM
Hi, We are using the getMongo processor to pull data from mongodb. Everything works fine but all dates are coming in the following format: "$numberLong" : "1542271444000" How can I get the date as a string or date.
... View more
Labels:
- Labels:
-
Apache NiFi
11-07-2018
01:23 PM
Hi, I have a clustered nifi setup and we are running GetMongo processor with the Primary mode on, so that duplicate data is not fetched. This seems to be working fine. However once I have this data I want the following processes in the chain to run on a cluster, as in parallel processing to be done on this data which has been fetched. Somehow this is not happening. So my question is below assuming GetMongo has fetched 30000 records and they are in the queue: 1) How do I check whether a processor is running its process on a single node or on all nodes. The config has been set to all nodes, but when the processor is running I see it displays 1 in the top right corner. 2) If one processor has been set to run only on primary node, do all other processors in the flow also run on Primary mode? Example: nifi.png In the screenshot above, my getmongo is running in primary node, how do I make sure that the execute script processor runs in parallel on all 3 nifi nodes. As of now if I check the view status history in the executescript process I see data flowing only through the primary node.
... View more
Labels:
- Labels:
-
Apache NiFi
11-05-2018
01:39 PM
Hi, I have a running cluster with 3 nifi nodes, all running an embedded zookeeper. Now I want to add a fourth node which doesn't run an embedded zookeeper. What I did was, on the new node edited the nifi properties, changed nifi.cluster.is.node to true, and added the first 3 nodes zookeeper url to nifi.zookeeper.connect.string. Once done I started nifi, however I get the below error. Posting the full stack trace, this stacktrace keeps on coming infinitely:
2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Below is a longer stack-trace right from startup till the error:
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures} 2018-11-05 06:09:41,961 INFO [main] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317 2018-11-05 06:09:44,086 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1763ms 2018-11-05 06:09:44,247 INFO [main] org.eclipse.jetty.server.session DefaultSessionIdManager workerName=node0 2018-11-05 06:09:44,248 INFO [main] org.eclipse.jetty.server.session No SessionScavenger set, using defaults 2018-11-05 06:09:44,249 INFO [main] org.eclipse.jetty.server.session Scavenging every 660000ms 2018-11-05 06:09:44,293 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5b49b1df{/nifi-update-attribute-ui-1.7.1,file:///nifi/work/jetty/nifi-update-attribute-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-update-attribute-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-update-attribute-ui-1.7.1.war} 2018-11-05 06:09:44,915 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=511ms 2018-11-05 06:09:44,963 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@11ec2b2f{/nifi-standard-content-viewer-1.7.1,file:///nifi/work/jetty/nifi-standard-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-standard-content-viewer-1.7.1.war} 2018-11-05 06:09:45,009 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=34ms 2018-11-05 06:09:45,041 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@2446be5f{/nifi-image-viewer-1.7.1,file:///nifi/work/jetty/nifi-image-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-media-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-image-viewer-1.7.1.war} 2018-11-05 06:09:46,731 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=1430ms 2018-11-05 06:09:46,770 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@71dfca65{/nifi-jolt-transform-json-ui-1.7.1,file:///nifi/work/jetty/nifi-jolt-transform-json-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-standard-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-jolt-transform-json-ui-1.7.1.war} 2018-11-05 06:09:46,959 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=38ms 2018-11-05 06:09:46,965 INFO [main] org.eclipse.jetty.ContextHandler./nifi No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:09:47,006 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1d6dc2b8{/nifi,file:///nifi/work/jetty/nifi-web-ui-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-ui-1.7.1.war} 2018-11-05 06:09:47,124 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=57ms 2018-11-05 06:09:47,164 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:09:47,180 INFO [main] o.eclipse.jetty.ContextHandler./nifi-api Initializing Spring root WebApplicationContext 2018-11-05 06:09:49,601 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties' 2018-11-05 06:09:49,602 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/nifi/./conf/nifi.properties' 2018-11-05 06:09:49,603 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 148 properties from /nifi/./conf/nifi.properties 2018-11-05 06:09:51,129 INFO [main] o.a.n.admin.AuditDataSourceFactoryBean Database not built for repository: jdbc:h2:./database_repository/nifi-flow-audit;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE.Building now... 2018-11-05 06:09:51,437 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 94 properties from system properties and environment variables 2018-11-05 06:09:51,438 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 94 properties.Including precedence overrides effective accessible registry key size is 94 2018-11-05 06:09:51,485 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository using 256 partitions 2018-11-05 06:09:51,507 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 0 records 2018-11-05 06:09:51,527 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 0 2018-11-05 06:09:51,533 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 5424567582 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2018-11-05 06:09:51,534 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2018-11-05 06:09:51,657 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures} 2018-11-05 06:09:51,671 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5eb054 checkpointed with 0 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1 2018-11-05 06:09:51,755 INFO [main] o.apache.nifi.controller.FlowController Checking if there is already a Cluster Coordinator Elected... 2018-11-05 06:09:51,826 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2018-11-05 06:10:00,725 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected 2018-11-05 06:10:00,727 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting 2018-11-05 06:10:01,832 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role. 2018-11-05 06:10:01,835 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2018-11-05 06:10:01,836 INFO [main] o.a.c.f.imps.CuratorFrameworkImpl Starting 2018-11-05 06:10:01,845 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Registered new Leader Selector for role Cluster Coordinator; this node is an active participant in the election. 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] started 2018-11-05 06:10:01,846 INFO [main] o.a.n.c.c.h.AbstractHeartbeatMonitor Heartbeat Monitor started 2018-11-05 06:10:04,629 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6149184e{/nifi-api,file:///nifi/work/jetty/nifi-web-api-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.7.1.war} 2018-11-05 06:10:05,255 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=463ms 2018-11-05 06:10:05,287 INFO [main] o.e.j.C./nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:10:05,290 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@8eb1297{/nifi-content-viewer,file:///nifi/work/jetty/nifi-web-content-viewer-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.7.1.war} 2018-11-05 06:10:05,323 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=20ms 2018-11-05 06:10:05,325 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default 2018-11-05 06:10:05,367 INFO [main] o.e.jetty.ContextHandler./nifi-docs No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:10:05,370 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@646ba7be{/nifi-docs,file:///nifi/work/jetty/nifi-web-docs-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.7.1.war} 2018-11-05 06:10:05,421 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=35ms 2018-11-05 06:10:05,471 INFO [main] org.eclipse.jetty.ContextHandler./ No Spring WebApplicationInitializer types detected on classpath 2018-11-05 06:10:05,475 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@1b6dab22{/,file:///nifi/work/jetty/nifi-web-error-1.7.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.7.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.7.1.war} 2018-11-05 06:10:05,492 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2a654167{HTTP/1.1,[http/1.1]}{<node-ip>:8081} 2018-11-05 06:10:05,492 INFO [main] org.eclipse.jetty.server.Server Started @47964ms 2018-11-05 06:10:05,505 INFO [main] org.apache.nifi.web.server.JettyServer Loading Flow... 2018-11-05 06:10:05,509 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 8082 2018-11-05 06:10:05,587 INFO [main] o.apache.nifi.controller.FlowController Successfully synchronized controller with proposed flow 2018-11-05 06:10:05,610 INFO [main] o.a.nifi.controller.StandardFlowService Connecting Node:<node-ip>:8081 2018-11-05 06:10:13,959 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2018-11-05 06:10:13,959 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2018-11-05 06:10:21,670 WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again 2018-11-05 06:10:21,670 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered 2018-11-05 06:10:21,675 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED 2018-11-05 06:10:21,677 INFO [Curator-ConnectionStateManager-0] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1d2e0826 Connection State changed to SUSPENDED p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures} 2018-11-05 06:10:21,685 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave up org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728) at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:857) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-11-05 06:10:21,687 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache NiFi
11-02-2018
12:16 PM
@ashok.kumar: I folllowed the above steps, i.e just adding the zookeper string and enabling is cluser in nifi properties. When I start the server I get the below error at a very fast rate: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.Apple-tab-span {white-space:pre} 2018-11-02 09:22:09,009 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:838) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Any pointers?
... View more
11-02-2018
10:20 AM
Version is not an issue, got the same issue when using 1.7.1 consume kafka and 1.7.1 nifi
... View more
11-02-2018
09:34 AM
@Rafeeq Shanavaz: My nifi version is 1.7.1 and consume kafka version is 0.10.1.7.1, not sure whether they are compatible or not. Can this be an issue?
... View more
11-02-2018
09:08 AM
Hi, We have a cluster running with 6 nodes. Now when I add a Kafka consumer, each cluster node should pull unique data, as in each node should fetch from a diff partition: https://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka. The same is also mentioned in the nifi docs. However in our case each node is pulling the same data from Kafka leading to duplication. Can you please help. Are there any specific configurations required to get the same done?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
10-31-2018
03:57 AM
@ashok.kumar: Thanks Ashok, my query is answered. On another note ideally in how many nodes should I keep the embedded zookeeper server running. If I have a six node cluster, in how many should I keep the embedded zookeeper on.
... View more
10-30-2018
10:26 AM
@ashok.kumar: Got it, thanks, just one last thing. All online tutorial require us to edit the zookeeper.conf file with some thing like: <node-ip>:2888:3888, what changes did you make with regard to this file?
... View more
10-30-2018
09:39 AM
@ashok.kumar Just to confirm: You edited the nifi.zookeeper.connect.string in the existing 2 node's nifi.properties file and also added the same in the new node. Then you would have edited the zookeeper.conf on the 2 existing boxes and also on the new box Lastly you would have started nifi on the new node. This made it work?
... View more
10-30-2018
07:49 AM
Hi, We have a 3 node nifi cluster running, using embedded Zookeeper. Now I need to add a new node to the cluster. What is the procedure for the same? Will I have to bring down all the nodes, since I need to add an entry to the /nifi/conf/zookeeper.conf(server=) and also to the list of zookeeper servers in /nifi/conf/nifi.properties(nifi.zookeeper.connect.string). As far as I understand, these both will have to be edited on the existing 3 as well as the new box and then the nifi server restarted. But dont think this is the right way, since each time the cluster will have to be brought down. Can you please advice?
... View more
Labels:
- Labels:
-
Apache NiFi